[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:17:47 dist-test-slave-r81b krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:17:47 dist-test-slave-r81b krb5kdc[399](info): set up 2 sockets
Jun 20 00:17:47 dist-test-slave-r81b krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260620 00:17:48.036437 392 mini_kdc.cc:121] Time spent starting KDC: real 1.097s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:17:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:17:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:17:48.159847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44097
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44097
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:48.242424 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:48.242651 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:48.242674 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:48.243902 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:17:48.243954 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:48.243968 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:48.243981 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:17:48.243993 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:17:48.245635 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:48.245923 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:48.246193 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:48.251911 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:48.251976 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:48.251933 412 server_base.cc:1034] running on GCE node
W20260620 00:17:48.251916 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:48.252390 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:48.252708 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:48.253898 412 hybrid_clock.cc:648] HybridClock initialized: now 1781914668253879 us; error 62 us; skew 500 ppm
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:48.256649 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:17:48.257145 412 webserver.cc:460] Webserver started at http://127.0.98.62:39095/ using document root <none> and password file <none>
I20260620 00:17:48.257382 412 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:48.257452 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:48.257601 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:48.258678 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "f56a9bc873994cecabeb1057abe3465b"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "2d8f8896fe0c55e9cf74e1958d34aa44"
server_key_iv: "ce2d3a59e98f6be71a1e9a19612c336f"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.259047 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "f56a9bc873994cecabeb1057abe3465b"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "2d8f8896fe0c55e9cf74e1958d34aa44"
server_key_iv: "ce2d3a59e98f6be71a1e9a19612c336f"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.260526 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:17:48.261475 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.261744 412 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:48.261878 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "f56a9bc873994cecabeb1057abe3465b"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "2d8f8896fe0c55e9cf74e1958d34aa44"
server_key_iv: "ce2d3a59e98f6be71a1e9a19612c336f"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.261998 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:48.270709 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:48.273351 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:48.273545 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:48.277721 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44097
I20260620 00:17:48.277750 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44097 every 8 connection(s)
I20260620 00:17:48.278122 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:17:48.279129 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:48.281477 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b: Bootstrap starting.
I20260620 00:17:48.282047 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:48.282305 480 log.cc:824] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:48.283068 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b: No bootstrap required, opened a new log
I20260620 00:17:48.284736 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a9bc873994cecabeb1057abe3465b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260620 00:17:48.284874 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:48.284896 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f56a9bc873994cecabeb1057abe3465b, State: Initialized, Role: FOLLOWER
I20260620 00:17:48.285045 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 412
I20260620 00:17:48.285061 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a9bc873994cecabeb1057abe3465b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260620 00:17:48.285137 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:17:48.285159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:17:48.285183 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:17:48.285235 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:48.285790 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a9bc873994cecabeb1057abe3465b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260620 00:17:48.285809 392 external_mini_cluster.cc:1428] Setting key 07a5a2bcd4267fc3e55ecbbfa71e806e
I20260620 00:17:48.285873 480 leader_election.cc:302] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f56a9bc873994cecabeb1057abe3465b; no voters:
I20260620 00:17:48.286078 480 leader_election.cc:288] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:17:48.286124 483 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:48.286307 483 raft_consensus.cc:686] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [term 1 LEADER]: Becoming Leader. State: Replica: f56a9bc873994cecabeb1057abe3465b, State: Running, Role: LEADER
I20260620 00:17:48.286350 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:17:48.286422 483 consensus_queue.cc:237] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a9bc873994cecabeb1057abe3465b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } }
I20260620 00:17:48.286926 484 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f56a9bc873994cecabeb1057abe3465b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a9bc873994cecabeb1057abe3465b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } } }
I20260620 00:17:48.287001 484 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:48.287266 485 sys_catalog.cc:455] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f56a9bc873994cecabeb1057abe3465b. Latest consensus state: current_term: 1 leader_uuid: "f56a9bc873994cecabeb1057abe3465b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f56a9bc873994cecabeb1057abe3465b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44097 } } }
I20260620 00:17:48.287395 485 sys_catalog.cc:458] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:48.287818 494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:17:48.288535 494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:17:48.290196 494 catalog_manager.cc:1347] Generated new cluster ID: 447a519df3054d80a1c698ac79531d45
I20260620 00:17:48.290252 494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:17:48.298189 494 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:17:48.298710 494 catalog_manager.cc:1504] Loading token signing keys...
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:48.301671 494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f56a9bc873994cecabeb1057abe3465b: Generated new TSK 0
I20260620 00:17:48.301877 494 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:17:48.304385 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:48.286776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33838 (local address 127.0.98.62:44097)
0620 00:17:48.287526 (+ 750us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:48.287536 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:48.287574 (+ 38us) server_negotiation.cc:407] Connection header received
0620 00:17:48.289828 (+ 2254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:48.289835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:48.289891 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:48.290030 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:48.292177 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.293612 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.295151 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.295752 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.295856 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:48.300277 (+ 4421us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:48.300282 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:48.300282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:48.300294 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:48.302395 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.302899 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.302902 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.302902 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.303001 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.303295 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.303295 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.303296 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.303553 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:48.303654 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:48.303974 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 00:17:48.304095 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":69,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:17:48.331311 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:48.416282 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:48.416527 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:48.416569 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:48.417865 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:48.417941 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:48.417990 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:17:48.419598 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:48.419895 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:48.420192 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:48.426903 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:48.426887 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:48.426887 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:48.427172 506 server_base.cc:1034] running on GCE node
I20260620 00:17:48.427492 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:48.427845 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:48.429024 506 hybrid_clock.cc:648] HybridClock initialized: now 1781914668429004 us; error 42 us; skew 500 ppm
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:48.432495 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:17:48.433001 506 webserver.cc:460] Webserver started at http://127.0.98.1:37507/ using document root <none> and password file <none>
I20260620 00:17:48.433261 506 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:48.433336 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:48.433486 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:48.435045 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "e2c2882d8bf4425e87ec308f431dd42b"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "3cc02dbdad1b0607a058627e1d8208cf"
server_key_iv: "59d5b4941926ad69c825bdb438445cc0"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.435429 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e2c2882d8bf4425e87ec308f431dd42b"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "3cc02dbdad1b0607a058627e1d8208cf"
server_key_iv: "59d5b4941926ad69c825bdb438445cc0"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.436973 506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260620 00:17:48.437824 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.437997 506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:48.438122 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "e2c2882d8bf4425e87ec308f431dd42b"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "3cc02dbdad1b0607a058627e1d8208cf"
server_key_iv: "59d5b4941926ad69c825bdb438445cc0"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.438202 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:48.458017 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:48.460611 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:48.460775 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:48.461057 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:17:48.461386 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:48.461419 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.461443 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:48.461455 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.468186 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38161
I20260620 00:17:48.468236 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38161 every 8 connection(s)
I20260620 00:17:48.468626 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:48.477416 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 506
I20260620 00:17:48.477530 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:17:48.477717 392 external_mini_cluster.cc:1428] Setting key 16ea079787312c2d8a72485437a822e5
I20260620 00:17:48.480530 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:48.469558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56563 (local address 127.0.98.62:44097)
0620 00:17:48.469621 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:48.469623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:48.470184 (+ 561us) server_negotiation.cc:407] Connection header received
0620 00:17:48.470680 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:48.470681 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:48.470693 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:48.470775 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:48.471810 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.472431 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.473571 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.473822 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.473912 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:48.476475 (+ 2563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:48.476477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:48.476477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:48.476485 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:48.478405 (+ 1920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.479026 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.479028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.479028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.479104 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.479487 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.479488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.479489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.479639 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:48.479705 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:48.480230 (+ 525us) server_negotiation.cc:299] Negotiation successful
0620 00:17:48.480367 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:17:48.480576 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:48.469428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44097 (local address 127.0.98.1:56563)
0620 00:17:48.469621 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:48.469666 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:48.470566 (+ 900us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:48.470886 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:48.470890 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:48.471504 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:48.471705 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:48.471709 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.472568 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.472569 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:48.473445 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:48.473447 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.473941 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.473942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:48.474071 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:48.474684 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:48.474693 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:48.476348 (+ 1655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:48.478495 (+ 2147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:48.478503 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:48.478506 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:48.478930 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:48.479182 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:48.479183 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:48.479184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:48.479353 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:48.479721 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:48.479728 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:48.479926 (+ 198us) client_negotiation.cc:769] Sending connection context
0620 00:17:48.480066 (+ 140us) client_negotiation.cc:240] Negotiation successful
0620 00:17:48.480324 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":114,"threads_started":1}
I20260620 00:17:48.481607 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44097
I20260620 00:17:48.481849 635 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:48.482076 635 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260620 00:17:48.482784 444 ts_manager.cc:194] Registered new tserver with Master: e2c2882d8bf4425e87ec308f431dd42b (127.0.98.1:38161)
I20260620 00:17:48.483850 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56563
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:17:48.507354 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:48.595428 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:48.595670 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:48.595701 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:48.596865 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:48.596912 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:48.596948 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:17:48.598369 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:48.598567 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:48.598758 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:48.604020 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:48.604171 640 server_base.cc:1034] running on GCE node
W20260620 00:17:48.604177 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:48.604013 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:48.604591 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:48.604856 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:48.606034 640 hybrid_clock.cc:648] HybridClock initialized: now 1781914668606006 us; error 39 us; skew 500 ppm
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:48.608938 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:17:48.609490 640 webserver.cc:460] Webserver started at http://127.0.98.2:43011/ using document root <none> and password file <none>
I20260620 00:17:48.609730 640 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:48.609802 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:48.609959 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:48.611037 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "5ce495b917384650bcb20f0b521e6bc8"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "5f827286bdcf3010dca3d8d5d6986ca8"
server_key_iv: "5e6fdd3b3ec4186c9bf0e55867afe464"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.611400 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5ce495b917384650bcb20f0b521e6bc8"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "5f827286bdcf3010dca3d8d5d6986ca8"
server_key_iv: "5e6fdd3b3ec4186c9bf0e55867afe464"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.612900 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:17:48.613837 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.614048 640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:48.614161 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "5ce495b917384650bcb20f0b521e6bc8"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "5f827286bdcf3010dca3d8d5d6986ca8"
server_key_iv: "5e6fdd3b3ec4186c9bf0e55867afe464"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.614238 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:48.629994 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:48.632566 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:48.632746 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:48.632969 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:17:48.633342 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:48.633401 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.633450 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:48.633487 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.639669 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35813
I20260620 00:17:48.639712 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35813 every 8 connection(s)
I20260620 00:17:48.640053 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:17:48.642943 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 640
I20260620 00:17:48.643083 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:17:48.643260 392 external_mini_cluster.cc:1428] Setting key 75a858ac97e51a3af689f2fffcb24682
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:17:48.651749 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:48.640901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57927 (local address 127.0.98.62:44097)
0620 00:17:48.641012 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:48.641014 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:48.641716 (+ 702us) server_negotiation.cc:407] Connection header received
0620 00:17:48.642345 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:48.642346 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:48.642359 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:48.642435 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:48.643733 (+ 1298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.644358 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.645839 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.646053 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.646131 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:48.648645 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:48.648647 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:48.648647 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:48.648655 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:48.650227 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.650680 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.650681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.650682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.650745 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.651009 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.651010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.651010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.651134 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:48.651211 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:48.651565 (+ 354us) server_negotiation.cc:299] Negotiation successful
0620 00:17:48.651649 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260620 00:17:48.651968 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:48.640863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44097 (local address 127.0.98.2:57927)
0620 00:17:48.641104 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:48.641144 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:48.642158 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:48.642560 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:48.642564 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:48.643367 (+ 803us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:48.643642 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:48.643648 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.644444 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.644445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:48.645733 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:48.645736 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.646136 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.646137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:48.646269 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:48.646859 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:48.646868 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:48.648478 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:48.650347 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:48.650355 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:48.650358 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:48.650598 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:48.650814 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:48.650815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:48.650815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:48.650926 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:48.651203 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:48.651208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:48.651367 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 00:17:48.651543 (+ 176us) client_negotiation.cc:240] Negotiation successful
0620 00:17:48.651717 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":118,"threads_started":1}
I20260620 00:17:48.652453 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44097
I20260620 00:17:48.652581 769 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:48.652784 769 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260620 00:17:48.653338 444 ts_manager.cc:194] Registered new tserver with Master: 5ce495b917384650bcb20f0b521e6bc8 (127.0.98.2:35813)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 00:17:48.653908 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:17:48.669358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:48.755301 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:48.755548 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:48.755575 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:48.756896 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:48.756946 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:48.756979 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:17:48.758419 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44097
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:48.758635 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:48.758826 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:48.764066 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:48.764118 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:48.764074 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:48.764074 774 server_base.cc:1034] running on GCE node
I20260620 00:17:48.764585 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:48.764896 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:48.766085 774 hybrid_clock.cc:648] HybridClock initialized: now 1781914668766065 us; error 54 us; skew 500 ppm
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:48.769157 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:17:48.769613 774 webserver.cc:460] Webserver started at http://127.0.98.3:40535/ using document root <none> and password file <none>
I20260620 00:17:48.769855 774 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:48.769901 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:48.770047 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:48.771184 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "3321bcf79fcc4b429e44f10620c8d61e"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "9b487b874ac1706f38adcaa1bb2c967a"
server_key_iv: "972a81f9c97a0fec0e8edaeb1e875996"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.771557 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3321bcf79fcc4b429e44f10620c8d61e"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "9b487b874ac1706f38adcaa1bb2c967a"
server_key_iv: "972a81f9c97a0fec0e8edaeb1e875996"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.773140 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:17:48.774080 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.774428 774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:48.774572 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "3321bcf79fcc4b429e44f10620c8d61e"
format_stamp: "Formatted at 2026-06-20 00:17:48 on dist-test-slave-r81b"
server_key: "9b487b874ac1706f38adcaa1bb2c967a"
server_key_iv: "972a81f9c97a0fec0e8edaeb1e875996"
server_key_version: "encryptionkey@0"
I20260620 00:17:48.774669 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:48.796985 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:48.799419 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:48.799539 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:48.799741 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:17:48.800061 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:48.800113 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.800186 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:48.800215 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:48.807582 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46323
I20260620 00:17:48.807684 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46323 every 8 connection(s)
I20260620 00:17:48.808013 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:48.815227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 774
I20260620 00:17:48.815318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:17:48.815483 392 external_mini_cluster.cc:1428] Setting key b16251ad60eb5a451287e08b9106bc50
I20260620 00:17:48.819111 487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:48.808970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52033 (local address 127.0.98.62:44097)
0620 00:17:48.809062 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:48.809065 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:48.809281 (+ 216us) server_negotiation.cc:407] Connection header received
0620 00:17:48.809960 (+ 679us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:48.809961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:48.809973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:48.810038 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:48.811082 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.811601 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.812822 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.813018 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.813078 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:48.815619 (+ 2541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:48.815621 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:48.815621 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:48.815631 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:48.817258 (+ 1627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.817850 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.817852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.817852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.817937 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.818339 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.818340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.818340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.818516 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:48.818586 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:48.818961 (+ 375us) server_negotiation.cc:299] Negotiation successful
0620 00:17:48.819040 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 00:17:48.819262 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:48.808903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44097 (local address 127.0.98.3:52033)
0620 00:17:48.809133 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:48.809179 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:48.809782 (+ 603us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:48.810119 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:48.810123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:48.810767 (+ 644us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:48.810975 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:48.810980 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.811745 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.811746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:48.812735 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:48.812738 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.813096 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.813097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:48.813210 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:48.813822 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:48.813830 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:48.815488 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:48.817386 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:48.817392 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:48.817394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:48.817721 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:48.818016 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:48.818016 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:48.818017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:48.818137 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:48.818627 (+ 490us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:48.818633 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:48.818779 (+ 146us) client_negotiation.cc:769] Sending connection context
0620 00:17:48.818896 (+ 117us) client_negotiation.cc:240] Negotiation successful
0620 00:17:48.819047 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":146,"threads_started":1}
I20260620 00:17:48.819811 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44097
I20260620 00:17:48.819941 903 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:48.820220 903 heartbeater.cc:511] Master 127.0.98.62:44097 requested a full tablet report, sending...
I20260620 00:17:48.820732 444 ts_manager.cc:194] Registered new tserver with Master: 3321bcf79fcc4b429e44f10620c8d61e (127.0.98.3:46323)
I20260620 00:17:48.821130 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52033
I20260620 00:17:48.827725 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:17:48.853039 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:48.844447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46490 (local address 127.0.98.1:38161)
0620 00:17:48.844673 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:48.844679 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:48.844693 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:17:48.844729 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:48.844731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:48.844746 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:48.844824 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:48.845427 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.846217 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.847011 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.847375 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.847440 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:48.849925 (+ 2485us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:48.849936 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:48.849936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:48.849944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:48.851714 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.852076 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.852079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.852080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.852194 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.852403 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.852403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.852404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.852624 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:48.852759 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:48.852874 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 00:17:48.852944 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":160,"threads_started":1}
W20260620 00:17:48.853473 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46490
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:17:48.877910 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:48.870357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46494 (local address 127.0.98.1:38161)
0620 00:17:48.870422 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:48.870424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:48.870457 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:17:48.870533 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:48.870534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:48.870546 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:48.870597 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:48.871141 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.871636 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.872476 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:48.872726 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:48.872794 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:48.875274 (+ 2480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:48.875275 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:48.875276 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:48.875283 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:48.876871 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.877193 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.877194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.877194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.877245 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:48.877455 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:48.877455 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:48.877456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:48.877608 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:48.877700 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:48.877780 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 00:17:48.877842 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:17:48.878484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 506
I20260620 00:17:48.883332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 640
I20260620 00:17:48.889262 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 774
I20260620 00:17:48.894053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 412
2026-06-20T00:17:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1983 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[917](info): set up 2 sockets
Jun 20 00:17:48 dist-test-slave-r81b krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260620 00:17:49.971964 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:17:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:17:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:17:50.091600 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34163
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34163
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:50.177021 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:50.177238 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:50.177263 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:50.178462 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:17:50.178637 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:50.178709 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:50.178731 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:17:50.178747 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:17:50.180619 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34163
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34163
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:50.180898 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:50.181146 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:50.186286 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:50.186311 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:50.186293 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:50.186609 930 server_base.cc:1034] running on GCE node
I20260620 00:17:50.186892 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:50.187167 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:50.188309 930 hybrid_clock.cc:648] HybridClock initialized: now 1781914670188285 us; error 38 us; skew 500 ppm
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:50.190814 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:17:50.191250 930 webserver.cc:460] Webserver started at http://127.0.98.62:39947/ using document root <none> and password file <none>
I20260620 00:17:50.191491 930 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:50.191540 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:50.191691 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:50.192848 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "f4a80cbd91c2466f9dc120d09b1eac73"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "a05856122be8ca16be6b77220137f9c3"
server_key_iv: "860d31c4e02af8852e97068daf87af29"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.193235 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "f4a80cbd91c2466f9dc120d09b1eac73"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "a05856122be8ca16be6b77220137f9c3"
server_key_iv: "860d31c4e02af8852e97068daf87af29"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.194691 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:17:50.195574 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.195824 930 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:50.195959 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "f4a80cbd91c2466f9dc120d09b1eac73"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "a05856122be8ca16be6b77220137f9c3"
server_key_iv: "860d31c4e02af8852e97068daf87af29"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.196048 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:50.223778 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:50.226212 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:50.226382 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:50.230393 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34163
I20260620 00:17:50.230445 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34163 every 8 connection(s)
I20260620 00:17:50.230809 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:17:50.231669 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:50.233963 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73: Bootstrap starting.
I20260620 00:17:50.234483 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:50.234730 998 log.cc:824] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:50.235368 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73: No bootstrap required, opened a new log
I20260620 00:17:50.236996 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a80cbd91c2466f9dc120d09b1eac73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260620 00:17:50.237149 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:50.237174 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4a80cbd91c2466f9dc120d09b1eac73, State: Initialized, Role: FOLLOWER
I20260620 00:17:50.237267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 930
I20260620 00:17:50.237367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:17:50.237329 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a80cbd91c2466f9dc120d09b1eac73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260620 00:17:50.237438 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:17:50.237480 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:17:50.237550 392 external_mini_cluster.cc:1428] Setting key 8a727c3801c2e03c94415d082b1dd3e9
I20260620 00:17:50.237529 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:50.238206 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a80cbd91c2466f9dc120d09b1eac73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260620 00:17:50.238318 998 leader_election.cc:302] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4a80cbd91c2466f9dc120d09b1eac73; no voters:
I20260620 00:17:50.238502 998 leader_election.cc:288] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:17:50.238703 1003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:50.238806 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:17:50.238806 1003 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [term 1 LEADER]: Becoming Leader. State: Replica: f4a80cbd91c2466f9dc120d09b1eac73, State: Running, Role: LEADER
I20260620 00:17:50.238958 1003 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a80cbd91c2466f9dc120d09b1eac73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } }
I20260620 00:17:50.239305 1005 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4a80cbd91c2466f9dc120d09b1eac73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a80cbd91c2466f9dc120d09b1eac73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } } }
I20260620 00:17:50.239380 1005 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:50.239578 1004 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4a80cbd91c2466f9dc120d09b1eac73. Latest consensus state: current_term: 1 leader_uuid: "f4a80cbd91c2466f9dc120d09b1eac73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4a80cbd91c2466f9dc120d09b1eac73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34163 } } }
I20260620 00:17:50.239626 1012 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:17:50.239658 1004 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73 [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:50.240175 1012 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:17:50.241716 1012 catalog_manager.cc:1347] Generated new cluster ID: 4cafd2239ab7495282b645ffe64ac382
I20260620 00:17:50.241803 1012 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:50.251068 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.238073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38148 (local address 127.0.98.62:34163)
0620 00:17:50.238262 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.238270 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.238319 (+ 49us) server_negotiation.cc:407] Connection header received
0620 00:17:50.238453 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.238462 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.238513 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.238698 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:50.239619 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.241123 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.242112 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.242718 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.242820 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.246620 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:50.246625 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:50.246625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:50.246640 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:50.249359 (+ 2719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.249789 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.249792 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.249792 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.249900 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.250119 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.250120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.250120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.250423 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:50.250574 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.250693 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.250829 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":110,"threads_started":1}
I20260620 00:17:50.253644 1012 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:17:50.254158 1012 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:17:50.265796 1012 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4a80cbd91c2466f9dc120d09b1eac73: Generated new TSK 0
I20260620 00:17:50.266064 1012 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:17:50.296693 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:50.380330 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:50.380589 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:50.380631 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:50.381898 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:50.381978 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:50.382033 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:17:50.383643 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:50.383935 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:50.384228 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:50.389417 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:50.389410 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:50.389410 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:50.389734 1024 server_base.cc:1034] running on GCE node
I20260620 00:17:50.389956 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:50.390209 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:50.391387 1024 hybrid_clock.cc:648] HybridClock initialized: now 1781914670391375 us; error 36 us; skew 500 ppm
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:50.394111 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:17:50.394567 1024 webserver.cc:460] Webserver started at http://127.0.98.1:36501/ using document root <none> and password file <none>
I20260620 00:17:50.394822 1024 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:50.394892 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:50.395040 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:50.396121 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "05322f50fb884f71bcefa3a878818ad7"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "8f5e8cb949abbecce7dc24efd1c03e89"
server_key_iv: "e1123460b0eb4e6a441cfaec94081f18"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.396569 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "05322f50fb884f71bcefa3a878818ad7"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "8f5e8cb949abbecce7dc24efd1c03e89"
server_key_iv: "e1123460b0eb4e6a441cfaec94081f18"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.398032 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:17:50.399168 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.399405 1024 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:50.399523 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "05322f50fb884f71bcefa3a878818ad7"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "8f5e8cb949abbecce7dc24efd1c03e89"
server_key_iv: "e1123460b0eb4e6a441cfaec94081f18"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.399622 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:50.408450 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:50.410899 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:50.411087 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:50.411378 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:17:50.411754 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:50.411811 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.411854 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:50.411880 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.417779 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40875
I20260620 00:17:50.417863 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40875 every 8 connection(s)
I20260620 00:17:50.418203 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:17:50.422406 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 1024
I20260620 00:17:50.422487 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:17:50.422670 392 external_mini_cluster.cc:1428] Setting key a574a693638194e6cdf60ec5fbea14a3
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:50.428803 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.419028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58363 (local address 127.0.98.62:34163)
0620 00:17:50.419072 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.419074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.419286 (+ 212us) server_negotiation.cc:407] Connection header received
0620 00:17:50.419810 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.419811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.419827 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.419880 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:50.420970 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.421486 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.422572 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.422820 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.422908 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.425497 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:50.425500 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:50.425500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:50.425512 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:50.427205 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.427743 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.427745 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.427745 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.427806 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.428070 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.428071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.428071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.428249 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:50.428326 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.428618 (+ 292us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.428706 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:17:50.429031 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.419001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.1:58363)
0620 00:17:50.419140 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:50.419185 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:50.419701 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:50.419985 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:50.419989 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:50.420654 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:50.420855 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.420861 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.421585 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.421586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.422417 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.422419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.422919 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.422920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.423092 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.423673 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:50.423681 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:50.425371 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:50.427336 (+ 1965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:50.427343 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:50.427345 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:50.427646 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:50.427877 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:50.427878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:50.427878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:50.427991 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:50.428317 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:50.428324 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:50.428475 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 00:17:50.428597 (+ 122us) client_negotiation.cc:240] Negotiation successful
0620 00:17:50.428773 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
I20260620 00:17:50.429538 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260620 00:17:50.429636 1153 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:50.429834 1153 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260620 00:17:50.430361 962 ts_manager.cc:194] Registered new tserver with Master: 05322f50fb884f71bcefa3a878818ad7 (127.0.98.1:40875)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:17:50.431030 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:17:50.451391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:50.533501 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:50.533726 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:50.533751 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:50.534888 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:50.534929 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:50.534961 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:17:50.536543 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:50.536751 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:50.536931 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:50.542635 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:50.542618 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:50.542651 1158 server_base.cc:1034] running on GCE node
W20260620 00:17:50.542618 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:50.543098 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:50.543401 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:50.544643 1158 hybrid_clock.cc:648] HybridClock initialized: now 1781914670544627 us; error 33 us; skew 500 ppm
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:50.547259 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:17:50.547765 1158 webserver.cc:460] Webserver started at http://127.0.98.2:40637/ using document root <none> and password file <none>
I20260620 00:17:50.547977 1158 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:50.548019 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:50.548107 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:50.549201 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "947e52a87a31bf4bbe089d5ccf6ab1ee"
server_key_iv: "40108f8251a8d24363bbe9e0af96a198"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.549499 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "947e52a87a31bf4bbe089d5ccf6ab1ee"
server_key_iv: "40108f8251a8d24363bbe9e0af96a198"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.550860 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:17:50.551698 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.551911 1158 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:50.552017 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "947e52a87a31bf4bbe089d5ccf6ab1ee"
server_key_iv: "40108f8251a8d24363bbe9e0af96a198"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.552073 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:50.563386 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:50.565701 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:50.565821 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:50.566041 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:17:50.566396 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:50.566442 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.566466 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:50.566517 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.572089 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45209
I20260620 00:17:50.572232 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45209 every 8 connection(s)
I20260620 00:17:50.572531 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:17:50.576627 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 1158
I20260620 00:17:50.576751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:17:50.576946 392 external_mini_cluster.cc:1428] Setting key be547882501b95619422b776e5409bc4
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:50.582538 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.573357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33395 (local address 127.0.98.62:34163)
0620 00:17:50.573440 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.573442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.573668 (+ 226us) server_negotiation.cc:407] Connection header received
0620 00:17:50.574145 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.574146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.574158 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.574213 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:50.575142 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.575709 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.576832 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.577065 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.577136 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.579359 (+ 2223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:50.579361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:50.579361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:50.579369 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:50.581091 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.581516 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.581517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.581517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.581582 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.581818 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.581819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.581819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.581970 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:50.582050 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.582295 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.582391 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 00:17:50.582612 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.573319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.2:33395)
0620 00:17:50.573555 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:50.573592 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:50.574056 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:50.574269 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:50.574272 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:50.574835 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:50.575040 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.575045 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.575806 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.575807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.576735 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.576737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.577144 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.577145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.577295 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.577870 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:50.577878 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:50.579266 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:50.581186 (+ 1920us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:50.581194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:50.581196 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:50.581431 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:50.581652 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:50.581653 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:50.581653 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:50.581752 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:50.582042 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:50.582046 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:50.582163 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 00:17:50.582247 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:17:50.582371 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":141,"threads_started":1}
I20260620 00:17:50.583333 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260620 00:17:50.583446 1287 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:50.583648 1287 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260620 00:17:50.584092 962 ts_manager.cc:194] Registered new tserver with Master: ea185261d51e4ae98ac90aa6aa2e2fe8 (127.0.98.2:45209)
I20260620 00:17:50.584506 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33395
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:17:50.603890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:50.684962 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:50.685202 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:50.685228 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:50.686475 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:50.686533 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:50.686563 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:17:50.688081 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34163
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:50.688400 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:50.688679 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:50.694048 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:50.694240 1292 server_base.cc:1034] running on GCE node
W20260620 00:17:50.694048 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:50.694063 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:50.694607 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:50.694897 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:50.696092 1292 hybrid_clock.cc:648] HybridClock initialized: now 1781914670696067 us; error 44 us; skew 500 ppm
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:50.699148 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:17:50.699710 1292 webserver.cc:460] Webserver started at http://127.0.98.3:41109/ using document root <none> and password file <none>
I20260620 00:17:50.699994 1292 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:50.700047 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:50.700233 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:50.701439 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "954fcab95e1a40b7a6308dc4d544cf74"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "625f53374382d023b7d81e83f043d615"
server_key_iv: "69a385e89b73ebbe4e71aafd1aad1c01"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.701875 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "954fcab95e1a40b7a6308dc4d544cf74"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "625f53374382d023b7d81e83f043d615"
server_key_iv: "69a385e89b73ebbe4e71aafd1aad1c01"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.703557 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:17:50.704450 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.704773 1292 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:50.704898 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "954fcab95e1a40b7a6308dc4d544cf74"
format_stamp: "Formatted at 2026-06-20 00:17:50 on dist-test-slave-r81b"
server_key: "625f53374382d023b7d81e83f043d615"
server_key_iv: "69a385e89b73ebbe4e71aafd1aad1c01"
server_key_version: "encryptionkey@0"
I20260620 00:17:50.704981 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:50.732892 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:50.735121 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:50.735316 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:50.735587 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:17:50.735921 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:50.735980 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.736021 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:50.736050 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:50.742436 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35171
I20260620 00:17:50.742506 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35171 every 8 connection(s)
I20260620 00:17:50.742808 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:50.749203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 1292
I20260620 00:17:50.749307 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:17:50.749495 392 external_mini_cluster.cc:1428] Setting key 4875791d69a8fa099df234a9da69fc3f
I20260620 00:17:50.753103 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.743467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59533 (local address 127.0.98.62:34163)
0620 00:17:50.743510 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.743511 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.743741 (+ 230us) server_negotiation.cc:407] Connection header received
0620 00:17:50.744297 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.744299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.744309 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.744364 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:50.745299 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.745804 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.746870 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.747101 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.747174 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.749877 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:50.749880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:50.749880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:50.749890 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:50.751648 (+ 1758us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.752108 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.752109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.752109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.752228 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.752483 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.752484 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.752484 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.752606 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:50.752661 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.752957 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.753037 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:17:50.753328 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.743473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34163 (local address 127.0.98.3:59533)
0620 00:17:50.743622 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:50.743658 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:50.744118 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:50.744421 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:50.744425 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:50.745006 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:50.745202 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.745206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.745890 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.745890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.746766 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.746768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.747216 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.747217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.747330 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.747861 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:50.747870 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:50.749775 (+ 1905us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:50.751745 (+ 1970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:50.751751 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:50.751753 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:50.752017 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:50.752305 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:50.752306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:50.752306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:50.752416 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:50.752689 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:50.752695 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:50.752828 (+ 133us) client_negotiation.cc:769] Sending connection context
0620 00:17:50.752937 (+ 109us) client_negotiation.cc:240] Negotiation successful
0620 00:17:50.753075 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
I20260620 00:17:50.753821 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34163
I20260620 00:17:50.753938 1421 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:50.754130 1421 heartbeater.cc:511] Master 127.0.98.62:34163 requested a full tablet report, sending...
I20260620 00:17:50.754547 962 ts_manager.cc:194] Registered new tserver with Master: 954fcab95e1a40b7a6308dc4d544cf74 (127.0.98.3:35171)
I20260620 00:17:50.754985 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59533
I20260620 00:17:50.761492 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:17:50.769615 1002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.763073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38152 (local address 127.0.98.62:34163)
0620 00:17:50.763158 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.763160 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.763249 (+ 89us) server_negotiation.cc:407] Connection header received
0620 00:17:50.763350 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.763351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.763363 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.763420 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:50.764089 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.764683 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.765503 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.765717 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.765809 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.767043 (+ 1234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:50.767044 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:50.767045 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:50.767051 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:50.768660 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.768969 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.768970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.768970 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.769024 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.769182 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.769183 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.769183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.769325 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:50.769402 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.769488 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.769549 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 00:17:50.771941 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38152:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:17:50.773644 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:17:50.780339 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.776411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52194 (local address 127.0.98.1:40875)
0620 00:17:50.776600 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.776605 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.776796 (+ 191us) server_negotiation.cc:407] Connection header received
0620 00:17:50.776840 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.776842 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.776857 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.776923 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:50.777126 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.778219 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.779383 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.779907 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.779999 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.780062 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.780204 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.780244 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260620 00:17:50.780373 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.776164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35171 (local address 127.0.0.1:45854)
0620 00:17:50.776321 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:50.776362 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:50.776431 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:50.776685 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:50.776688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:50.776691 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:50.776755 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.776757 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.777887 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.777888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.779148 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.779150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.780093 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.780095 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.780188 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.780193 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:17:50.780224 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 00:17:50.780268 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":91,"threads_started":1}
I20260620 00:17:50.780368 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.776185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40875 (local address 127.0.0.1:52194)
0620 00:17:50.776687 (+ 502us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:50.776694 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:50.776736 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:50.777003 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:50.777003 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:50.777005 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:50.777053 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.777055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.778303 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.778304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.779283 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.779285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.780003 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.780004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.780081 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.780093 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 00:17:50.780192 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 00:17:50.780246 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"mutex_wait_us":2,"thread_start_us":109,"threads_started":1}
I20260620 00:17:50.780550 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.776164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45209 (local address 127.0.0.1:40024)
0620 00:17:50.776565 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:50.776575 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:50.776617 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:50.776873 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:50.776873 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:50.776875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:50.776912 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.776913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.778253 (+ 1340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.778254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.779357 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.779359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.780322 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.780323 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.780382 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.780386 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:50.780430 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:17:50.780472 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":3,"thread_start_us":49,"threads_started":1}
I20260620 00:17:50.780851 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.776541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.0.98.2:45209)
0620 00:17:50.776721 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.776726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.776736 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:17:50.776776 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.776778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.776793 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.776866 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:50.776968 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.778147 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.779428 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.780179 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.780239 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.780290 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.780743 (+ 453us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.780782 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":73,"threads_started":1}
I20260620 00:17:50.781210 1087 tablet_service.cc:1467] Processing CreateTablet for tablet 9c40a625b7534498bd18d459c6c5d59f (DEFAULT_TABLE table=test-table [id=ea2497ece9c54be19ad7da137cc13d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:50.781436 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.776331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45854 (local address 127.0.98.3:35171)
0620 00:17:50.776504 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.776509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.776520 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:17:50.776561 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.776564 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.776580 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.776664 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:50.777042 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.777773 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.779236 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.779990 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.781158 (+ 1168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.781287 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.781312 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.781355 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":100,"threads_started":1}
I20260620 00:17:50.781450 1221 tablet_service.cc:1467] Processing CreateTablet for tablet 9c40a625b7534498bd18d459c6c5d59f (DEFAULT_TABLE table=test-table [id=ea2497ece9c54be19ad7da137cc13d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:50.781608 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c40a625b7534498bd18d459c6c5d59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:50.781772 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c40a625b7534498bd18d459c6c5d59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:50.782008 1355 tablet_service.cc:1467] Processing CreateTablet for tablet 9c40a625b7534498bd18d459c6c5d59f (DEFAULT_TABLE table=test-table [id=ea2497ece9c54be19ad7da137cc13d10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:50.782303 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9c40a625b7534498bd18d459c6c5d59f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:50.784729 1438 tablet_bootstrap.cc:492] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7: Bootstrap starting.
I20260620 00:17:50.785348 1438 tablet_bootstrap.cc:654] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:50.785625 1439 tablet_bootstrap.cc:492] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8: Bootstrap starting.
I20260620 00:17:50.785959 1440 tablet_bootstrap.cc:492] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74: Bootstrap starting.
I20260620 00:17:50.785992 1438 log.cc:824] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:50.786425 1439 tablet_bootstrap.cc:654] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:50.786582 1440 tablet_bootstrap.cc:654] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:50.786743 1438 tablet_bootstrap.cc:492] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7: No bootstrap required, opened a new log
I20260620 00:17:50.786827 1438 ts_tablet_manager.cc:1397] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:17:50.787056 1439 log.cc:824] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:50.787118 1440 log.cc:824] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:50.787773 1439 tablet_bootstrap.cc:492] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8: No bootstrap required, opened a new log
I20260620 00:17:50.787817 1440 tablet_bootstrap.cc:492] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74: No bootstrap required, opened a new log
I20260620 00:17:50.788257 1439 ts_tablet_manager.cc:1397] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:17:50.788270 1440 ts_tablet_manager.cc:1397] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260620 00:17:50.789350 1438 raft_consensus.cc:348] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } }
I20260620 00:17:50.789539 1438 raft_consensus.cc:374] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:50.789575 1438 raft_consensus.cc:729] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05322f50fb884f71bcefa3a878818ad7, State: Initialized, Role: FOLLOWER
I20260620 00:17:50.789597 1440 raft_consensus.cc:348] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } }
I20260620 00:17:50.789695 1438 consensus_queue.cc:260] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } }
I20260620 00:17:50.789795 1440 raft_consensus.cc:374] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:50.789840 1440 raft_consensus.cc:729] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 954fcab95e1a40b7a6308dc4d544cf74, State: Initialized, Role: FOLLOWER
I20260620 00:17:50.789947 1440 consensus_queue.cc:260] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } }
I20260620 00:17:50.789971 1438 ts_tablet_manager.cc:1428] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:17:50.790141 1153 heartbeater.cc:503] Master 127.0.98.62:34163 was elected leader, sending a full tablet report...
I20260620 00:17:50.790246 1421 heartbeater.cc:503] Master 127.0.98.62:34163 was elected leader, sending a full tablet report...
I20260620 00:17:50.790232 1440 ts_tablet_manager.cc:1428] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:17:50.790215 1439 raft_consensus.cc:348] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } }
I20260620 00:17:50.790370 1439 raft_consensus.cc:374] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:50.790401 1439 raft_consensus.cc:729] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea185261d51e4ae98ac90aa6aa2e2fe8, State: Initialized, Role: FOLLOWER
I20260620 00:17:50.790526 1439 consensus_queue.cc:260] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } }
I20260620 00:17:50.790827 1287 heartbeater.cc:503] Master 127.0.98.62:34163 was elected leader, sending a full tablet report...
I20260620 00:17:50.791028 1439 ts_tablet_manager.cc:1428] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260620 00:17:50.823097 1288 tablet.cc:2369] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:17:50.837110 1446 raft_consensus.cc:482] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:17:50.837241 1446 raft_consensus.cc:504] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } }
I20260620 00:17:50.837620 1446 leader_election.cc:288] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 954fcab95e1a40b7a6308dc4d544cf74 (127.0.98.3:35171), 05322f50fb884f71bcefa3a878818ad7 (127.0.98.1:40875)
I20260620 00:17:50.841794 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.837902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33107 (local address 127.0.98.3:35171)
0620 00:17:50.837951 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.837953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.838086 (+ 133us) server_negotiation.cc:407] Connection header received
0620 00:17:50.838120 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.838121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.838134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.838191 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:50.838328 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.839127 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.840754 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.841377 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.841478 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.841495 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.841670 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.841718 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:17:50.841794 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.837893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38157 (local address 127.0.98.1:40875)
0620 00:17:50.837940 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.837943 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.837955 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:17:50.838024 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.838025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.838036 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.838131 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:50.838329 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.839121 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.840753 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.841369 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.841515 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.841528 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.841686 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.841728 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:17:50.841781 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.837822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40875 (local address 127.0.98.2:38157)
0620 00:17:50.837883 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:50.837896 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:50.837940 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:50.838154 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:50.838155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:50.838157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:50.838250 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.838252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.839209 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.839209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.840641 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.840644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.841514 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.841515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.841587 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.841604 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 00:17:50.841654 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:17:50.841682 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":59}
I20260620 00:17:50.841787 1447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.837822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35171 (local address 127.0.98.2:33107)
0620 00:17:50.838012 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:50.838021 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:50.838069 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:50.838201 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:50.838202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:50.838203 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:50.838255 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.838256 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.839208 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.839209 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.840641 (+ 1432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:50.840644 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.841488 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.841489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:50.841562 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.841567 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:17:50.841627 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 00:17:50.841675 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":99,"thread_start_us":107,"threads_started":1}
I20260620 00:17:50.842227 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c40a625b7534498bd18d459c6c5d59f" candidate_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05322f50fb884f71bcefa3a878818ad7" is_pre_election: true
I20260620 00:17:50.842242 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c40a625b7534498bd18d459c6c5d59f" candidate_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954fcab95e1a40b7a6308dc4d544cf74" is_pre_election: true
I20260620 00:17:50.842401 1375 raft_consensus.cc:2454] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea185261d51e4ae98ac90aa6aa2e2fe8 in term 0.
I20260620 00:17:50.842401 1107 raft_consensus.cc:2454] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea185261d51e4ae98ac90aa6aa2e2fe8 in term 0.
I20260620 00:17:50.842693 1177 leader_election.cc:302] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 954fcab95e1a40b7a6308dc4d544cf74, ea185261d51e4ae98ac90aa6aa2e2fe8; no voters:
I20260620 00:17:50.842895 1446 raft_consensus.cc:2781] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:17:50.842999 1446 raft_consensus.cc:482] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:17:50.843035 1446 raft_consensus.cc:3037] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:50.843808 1446 raft_consensus.cc:504] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } }
I20260620 00:17:50.843969 1446 leader_election.cc:288] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [CANDIDATE]: Term 1 election: Requested vote from peers 954fcab95e1a40b7a6308dc4d544cf74 (127.0.98.3:35171), 05322f50fb884f71bcefa3a878818ad7 (127.0.98.1:40875)
I20260620 00:17:50.844162 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c40a625b7534498bd18d459c6c5d59f" candidate_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "954fcab95e1a40b7a6308dc4d544cf74"
I20260620 00:17:50.844163 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9c40a625b7534498bd18d459c6c5d59f" candidate_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05322f50fb884f71bcefa3a878818ad7"
I20260620 00:17:50.844242 1375 raft_consensus.cc:3037] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:50.844265 1107 raft_consensus.cc:3037] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:50.844964 1375 raft_consensus.cc:2454] T 9c40a625b7534498bd18d459c6c5d59f P 954fcab95e1a40b7a6308dc4d544cf74 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea185261d51e4ae98ac90aa6aa2e2fe8 in term 1.
I20260620 00:17:50.844993 1107 raft_consensus.cc:2454] T 9c40a625b7534498bd18d459c6c5d59f P 05322f50fb884f71bcefa3a878818ad7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea185261d51e4ae98ac90aa6aa2e2fe8 in term 1.
I20260620 00:17:50.845144 1177 leader_election.cc:302] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 954fcab95e1a40b7a6308dc4d544cf74, ea185261d51e4ae98ac90aa6aa2e2fe8; no voters:
I20260620 00:17:50.845230 1446 raft_consensus.cc:2781] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:50.845402 1446 raft_consensus.cc:686] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [term 1 LEADER]: Becoming Leader. State: Replica: ea185261d51e4ae98ac90aa6aa2e2fe8, State: Running, Role: LEADER
I20260620 00:17:50.845510 1446 consensus_queue.cc:237] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } }
I20260620 00:17:50.846453 961 catalog_manager.cc:5495] T 9c40a625b7534498bd18d459c6c5d59f P ea185261d51e4ae98ac90aa6aa2e2fe8 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea185261d51e4ae98ac90aa6aa2e2fe8 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "954fcab95e1a40b7a6308dc4d544cf74" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35171 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea185261d51e4ae98ac90aa6aa2e2fe8" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05322f50fb884f71bcefa3a878818ad7" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40875 } health_report { overall_health: UNKNOWN } } }
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:17:50.858860 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.850144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52196 (local address 127.0.98.1:40875)
0620 00:17:50.850243 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.850245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.850358 (+ 113us) server_negotiation.cc:407] Connection header received
0620 00:17:50.850441 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.850442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.850462 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.850541 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:50.851173 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.851810 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.852654 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.852845 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.852948 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.855346 (+ 2398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:50.855360 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:50.855360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:50.855368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:50.857601 (+ 2233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.857979 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.857991 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.857991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.858051 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.858250 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.858251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.858252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.858404 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:50.858621 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.858659 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.858741 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:17:50.882041 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.874637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52200 (local address 127.0.98.1:40875)
0620 00:17:50.874687 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.874689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.874763 (+ 74us) server_negotiation.cc:407] Connection header received
0620 00:17:50.874811 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.874812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.874821 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.874870 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:50.875433 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.875926 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.876725 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.876926 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.876985 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.879375 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:50.879377 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:50.879378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:50.879385 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:50.880960 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.881293 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.881294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.881294 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.881345 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.881530 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.881531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.881531 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.881631 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:50.881704 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.881900 (+ 196us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.881972 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:17:50.908003 1436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:50.898767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.0.98.1:40875)
0620 00:17:50.898872 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:50.898875 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:50.898999 (+ 124us) server_negotiation.cc:407] Connection header received
0620 00:17:50.899075 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:50.899077 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:50.899124 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:50.899207 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:50.900077 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.900640 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.901427 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:50.901712 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:50.901778 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:50.905061 (+ 3283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:50.905063 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:50.905063 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:50.905073 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:50.906571 (+ 1498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.906991 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.906992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.906992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.907047 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:50.907339 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:50.907340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:50.907341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:50.907527 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:50.907622 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:50.907766 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 00:17:50.907859 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
W20260620 00:17:50.908613 1067 tablet_service.cc:2908] Rejecting scan request for tablet 9c40a625b7534498bd18d459c6c5d59f: Uninitialized: clean time has not yet been initialized
I20260620 00:17:50.909638 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 1024
I20260620 00:17:50.915370 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 1158
I20260620 00:17:50.919870 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 1292
I20260620 00:17:50.924466 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 930
2026-06-20T00:17:50Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2028 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[1462](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[1462](info): set up 2 sockets
Jun 20 00:17:50 dist-test-slave-r81b krb5kdc[1462](info): commencing operation
krb5kdc: starting...
W20260620 00:17:51.967988 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:17:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:17:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:17:52.091236 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45083
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:52.178162 1475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:52.178546 1475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:52.178609 1475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:52.180019 1475 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:17:52.180099 1475 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 00:17:52.180124 1475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:52.180145 1475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:52.180192 1475 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:17:52.180219 1475 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:17:52.182319 1475 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45083
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1475
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:52.182617 1475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:52.182928 1475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:52.188196 1481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:52.188295 1483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:52.188195 1480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:52.188469 1475 server_base.cc:1034] running on GCE node
I20260620 00:17:52.188863 1475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:52.189225 1475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:52.190402 1475 hybrid_clock.cc:648] HybridClock initialized: now 1781914672190371 us; error 48 us; skew 500 ppm
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:52.199940 1475 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:17:52.200541 1475 webserver.cc:460] Webserver started at http://127.0.98.62:36347/ using document root <none> and password file <none>
I20260620 00:17:52.200814 1475 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:52.200886 1475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:52.201054 1475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:52.202297 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "7f1af99f853c45eabf6df87418e35494"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "cc62e544b39e289a7a2866f248d7e985"
server_key_iv: "70282c957d38195e2585fd475003792b"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.202687 1475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "7f1af99f853c45eabf6df87418e35494"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "cc62e544b39e289a7a2866f248d7e985"
server_key_iv: "70282c957d38195e2585fd475003792b"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.204267 1475 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:17:52.205226 1490 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:52.205577 1475 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:52.205694 1475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "7f1af99f853c45eabf6df87418e35494"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "cc62e544b39e289a7a2866f248d7e985"
server_key_iv: "70282c957d38195e2585fd475003792b"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.205793 1475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:52.243016 1475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:52.245333 1475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:52.245503 1475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:52.250375 1475 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45083
I20260620 00:17:52.250408 1552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45083 every 8 connection(s)
I20260620 00:17:52.250769 1475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:17:52.251456 1553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:52.253803 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494: Bootstrap starting.
I20260620 00:17:52.254420 1553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:52.254734 1553 log.cc:824] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:52.255435 1553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494: No bootstrap required, opened a new log
I20260620 00:17:52.257129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 1475
I20260620 00:17:52.257231 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:17:52.257376 392 external_mini_cluster.cc:1428] Setting key e648cf6e99b402b050024cd862fdc3af
I20260620 00:17:52.258074 1553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1af99f853c45eabf6df87418e35494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45083 } }
I20260620 00:17:52.258324 1553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:52.258378 1553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f1af99f853c45eabf6df87418e35494, State: Initialized, Role: FOLLOWER
I20260620 00:17:52.258522 1553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1af99f853c45eabf6df87418e35494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45083 } }
I20260620 00:17:52.258620 1553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:17:52.258664 1553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:17:52.258713 1553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [term 0 FOLLOWER]: Advancing to term 1
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:52.266806 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.257911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35226 (local address 127.0.98.62:45083)
0620 00:17:52.258110 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:52.258121 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:52.258176 (+ 55us) server_negotiation.cc:407] Connection header received
0620 00:17:52.258301 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:52.258308 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:52.258367 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:52.258512 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:52.259428 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.260420 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.261123 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.261514 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.261597 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.263830 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:52.263834 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:52.263834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:52.263844 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:52.265451 (+ 1607us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.265798 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.265800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.265800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.265882 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.266043 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.266044 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.266044 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.266269 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:52.266365 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:52.266449 (+ 84us) server_negotiation.cc:299] Negotiation successful
0620 00:17:52.266612 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":111,"threads_started":1}
I20260620 00:17:52.276971 1553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1af99f853c45eabf6df87418e35494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45083 } }
I20260620 00:17:52.277172 1553 leader_election.cc:302] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7f1af99f853c45eabf6df87418e35494; no voters:
I20260620 00:17:52.277395 1553 leader_election.cc:288] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:17:52.277472 1558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:52.277665 1553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:17:52.277669 1558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [term 1 LEADER]: Becoming Leader. State: Replica: 7f1af99f853c45eabf6df87418e35494, State: Running, Role: LEADER
I20260620 00:17:52.277889 1558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1af99f853c45eabf6df87418e35494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45083 } }
I20260620 00:17:52.278275 1560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f1af99f853c45eabf6df87418e35494. Latest consensus state: current_term: 1 leader_uuid: "7f1af99f853c45eabf6df87418e35494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1af99f853c45eabf6df87418e35494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45083 } } }
I20260620 00:17:52.278380 1560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:52.278264 1559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7f1af99f853c45eabf6df87418e35494" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7f1af99f853c45eabf6df87418e35494" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45083 } } }
I20260620 00:17:52.278462 1559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494 [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:52.278599 1567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:17:52.279168 1567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:17:52.280767 1567 catalog_manager.cc:1347] Generated new cluster ID: f6564babb7374006880e5383f7b3230a
I20260620 00:17:52.280817 1567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:17:52.290839 1567 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:17:52.291373 1567 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:17:52.294443 1567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494: Generated new TSK 0
I20260620 00:17:52.294705 1567 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
W20260620 00:17:52.512097 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.1: real 0.216s user 0.002s sys 0.001s
I20260620 00:17:52.512462 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45083
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:52.593129 1579 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:52.593364 1579 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:52.593389 1579 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:52.594648 1579 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:52.594694 1579 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:52.594715 1579 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:17:52.594733 1579 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:17:52.596313 1579 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1579
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:52.596578 1579 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:52.596808 1579 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:52.601941 1585 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:52.601944 1584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:52.601944 1587 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:52.602258 1579 server_base.cc:1034] running on GCE node
I20260620 00:17:52.602458 1579 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:52.602705 1579 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:52.603852 1579 hybrid_clock.cc:648] HybridClock initialized: now 1781914672603830 us; error 41 us; skew 500 ppm
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:52.606496 1579 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:17:52.606992 1579 webserver.cc:460] Webserver started at http://127.0.98.1:36357/ using document root <none> and password file <none>
I20260620 00:17:52.607189 1579 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:52.607230 1579 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:52.607319 1579 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:52.608404 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "0442db359cd3423286098ad608a4dedb"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "2e0d73b90a94ac82625843be04c4b657"
server_key_iv: "7be84483996710559bee39d719482548"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.608903 1579 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0442db359cd3423286098ad608a4dedb"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "2e0d73b90a94ac82625843be04c4b657"
server_key_iv: "7be84483996710559bee39d719482548"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.610309 1579 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:17:52.611059 1594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:52.611233 1579 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:17:52.611330 1579 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "0442db359cd3423286098ad608a4dedb"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "2e0d73b90a94ac82625843be04c4b657"
server_key_iv: "7be84483996710559bee39d719482548"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.611413 1579 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:52.621901 1579 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:52.623796 1579 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:52.623981 1579 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:52.624572 1579 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:52.624629 1579 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:52.624668 1579 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:52.624692 1579 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:52.631940 1579 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38115
I20260620 00:17:52.632248 1709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38115 every 8 connection(s)
I20260620 00:17:52.632396 1579 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:17:52.634734 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.624934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54163 (local address 127.0.98.62:45083)
0620 00:17:52.625001 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:52.625003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:52.625250 (+ 247us) server_negotiation.cc:407] Connection header received
0620 00:17:52.625719 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:52.625720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:52.625734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:52.625813 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:52.626523 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.627209 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.628679 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.628896 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.628989 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.631692 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:52.631694 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:52.631695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:52.631707 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:52.633332 (+ 1625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.633767 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.633768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.633768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.633826 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.634082 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.634083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.634083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.634207 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:52.634279 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:52.634562 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 00:17:52.634645 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:17:52.634925 1603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.624943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45083 (local address 127.0.98.1:54163)
0620 00:17:52.625116 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:52.625154 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:52.625629 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:52.625890 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:52.625894 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:52.626219 (+ 325us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:52.626412 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.626417 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.627303 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.627304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.628573 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.628576 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.629021 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.629022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.629153 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.629718 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:52.629734 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:52.631574 (+ 1840us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:52.633433 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.633439 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.633441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.633669 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.633898 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.633899 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.633899 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.634013 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.634282 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:52.634288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:52.634425 (+ 137us) client_negotiation.cc:769] Sending connection context
0620 00:17:52.634541 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 00:17:52.634693 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":76,"threads_started":1}
I20260620 00:17:52.635433 1710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45083
I20260620 00:17:52.635519 1710 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:52.635739 1710 heartbeater.cc:511] Master 127.0.98.62:45083 requested a full tablet report, sending...
I20260620 00:17:52.636330 1506 ts_manager.cc:194] Registered new tserver with Master: 0442db359cd3423286098ad608a4dedb (127.0.98.1:38115)
I20260620 00:17:52.637149 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 1579
I20260620 00:17:52.637202 1506 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54163
I20260620 00:17:52.637234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:17:52.637410 392 external_mini_cluster.cc:1428] Setting key 0427599320be86a8487269942eee9c7d
I20260620 00:17:52.642385 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.636553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54807 (local address 127.0.98.62:45083)
0620 00:17:52.636595 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:52.636597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:52.636721 (+ 124us) server_negotiation.cc:407] Connection header received
0620 00:17:52.636746 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:52.636746 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:52.636756 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:52.636817 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:52.637123 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.637773 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.638559 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.638764 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.638834 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.639869 (+ 1035us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:52.639870 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:52.639870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:52.639876 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:52.641254 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.641631 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.641632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.641632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.641685 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.641887 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.641888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.641888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.641997 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:52.642070 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:52.642223 (+ 153us) server_negotiation.cc:299] Negotiation successful
0620 00:17:52.642297 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:17:52.642400 1716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.636507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45083 (local address 127.0.98.1:54807)
0620 00:17:52.636646 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:52.636655 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:52.636712 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:52.636821 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:52.636822 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:52.636960 (+ 138us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:52.637031 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.637032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.637856 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.637857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.638475 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.638476 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.638833 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.638834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.638902 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.639362 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:52.639369 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:52.639781 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:52.641354 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.641356 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.641356 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.641557 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.641759 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.641760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.641760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.641823 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.642062 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:52.642063 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:52.642113 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 00:17:52.642199 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 00:17:52.642297 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:17:52.663434 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45083
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:52.744927 1719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:52.745151 1719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:52.745175 1719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:52.746256 1719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:52.746290 1719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:52.746311 1719 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:17:52.746328 1719 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:17:52.747869 1719 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1719
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:52.748116 1719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:52.748366 1719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:52.753321 1724 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:52.753361 1727 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:52.753334 1725 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:52.753539 1719 server_base.cc:1034] running on GCE node
I20260620 00:17:52.753734 1719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:52.753921 1719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:52.755057 1719 hybrid_clock.cc:648] HybridClock initialized: now 1781914672755045 us; error 30 us; skew 500 ppm
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:52.757620 1719 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:17:52.758082 1719 webserver.cc:460] Webserver started at http://127.0.98.2:42281/ using document root <none> and password file <none>
I20260620 00:17:52.758265 1719 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:52.758303 1719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:52.758392 1719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:52.759398 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "36b1413312f8428b9c1b7b0f9b61968e"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "1b6254dce17eac7dd88342601f9125f1"
server_key_iv: "1fe9e768262538b8c0463c0830f85cd8"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.759701 1719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "36b1413312f8428b9c1b7b0f9b61968e"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "1b6254dce17eac7dd88342601f9125f1"
server_key_iv: "1fe9e768262538b8c0463c0830f85cd8"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.761070 1719 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:17:52.761837 1734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:52.762039 1719 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:17:52.762172 1719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "36b1413312f8428b9c1b7b0f9b61968e"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "1b6254dce17eac7dd88342601f9125f1"
server_key_iv: "1fe9e768262538b8c0463c0830f85cd8"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.762256 1719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:52.779698 1719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:52.782094 1719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:52.782260 1719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:52.782853 1719 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:52.782905 1719 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:52.782938 1719 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:52.782959 1719 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:52.790382 1719 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33853
I20260620 00:17:52.790462 1849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33853 every 8 connection(s)
I20260620 00:17:52.790813 1719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:17:52.793226 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.783315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35899 (local address 127.0.98.62:45083)
0620 00:17:52.783379 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:52.783382 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:52.783623 (+ 241us) server_negotiation.cc:407] Connection header received
0620 00:17:52.784120 (+ 497us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:52.784122 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:52.784176 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:52.784255 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:52.785031 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.785538 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.786560 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.786795 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.786881 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.790179 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:52.790181 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:52.790181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:52.790191 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:52.791736 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.792205 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.792206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.792207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.792272 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.792534 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.792535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.792535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.792669 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:52.792743 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:52.793032 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 00:17:52.793123 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:17:52.793424 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.783316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45083 (local address 127.0.98.2:35899)
0620 00:17:52.783490 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:52.783531 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:52.784027 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:52.784339 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:52.784343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:52.784686 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:52.784922 (+ 236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.784927 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.785647 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.785648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.786465 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.786467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.786893 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.786894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.787063 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.787603 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:52.787611 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:52.790050 (+ 2439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:52.791843 (+ 1793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.791850 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.791852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.792086 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.792342 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.792343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.792343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.792463 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.792746 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:52.792753 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:52.792900 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 00:17:52.793011 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:17:52.793189 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260620 00:17:52.793875 1850 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45083
I20260620 00:17:52.793962 1850 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:52.794170 1850 heartbeater.cc:511] Master 127.0.98.62:45083 requested a full tablet report, sending...
I20260620 00:17:52.794591 1507 ts_manager.cc:194] Registered new tserver with Master: 36b1413312f8428b9c1b7b0f9b61968e (127.0.98.2:33853)
I20260620 00:17:52.795075 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35899
I20260620 00:17:52.798697 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 1719
I20260620 00:17:52.798790 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:17:52.798976 392 external_mini_cluster.cc:1428] Setting key 31487ef6cb548657f2a9684a35bb0fdb
I20260620 00:17:52.801386 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.795031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56787 (local address 127.0.98.62:45083)
0620 00:17:52.795066 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:52.795068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:52.795280 (+ 212us) server_negotiation.cc:407] Connection header received
0620 00:17:52.795325 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:52.795325 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:52.795335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:52.795386 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:52.796076 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.796598 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.797262 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.797434 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.797482 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.798327 (+ 845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:52.798328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:52.798328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:52.798334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:52.800200 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.800513 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.800514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.800514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.800574 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.800754 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.800755 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.800755 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.800951 (+ 196us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:52.801027 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:52.801204 (+ 177us) server_negotiation.cc:299] Negotiation successful
0620 00:17:52.801293 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:17:52.801420 1856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.795030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45083 (local address 127.0.98.2:56787)
0620 00:17:52.795195 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:52.795206 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:52.795272 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:52.795799 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:52.795800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:52.795950 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:52.795999 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.796001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.796680 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.796681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.797193 (+ 512us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.797194 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.797495 (+ 301us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.797495 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.797538 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.797916 (+ 378us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:52.797920 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:52.798243 (+ 323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:52.800277 (+ 2034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.800278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.800278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.800442 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.800650 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.800651 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.800651 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.800697 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.801026 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:52.801027 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:52.801087 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 00:17:52.801167 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:17:52.801281 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:17:52.824913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45083
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:52.909397 1859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:52.909726 1859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:52.909780 1859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:52.911625 1859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:52.911696 1859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:52.911734 1859 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:17:52.911772 1859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:17:52.913426 1859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:45083
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:52.913722 1859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:52.914012 1859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:52.919392 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:52.919409 1867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:52.919385 1864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:52.919713 1859 server_base.cc:1034] running on GCE node
I20260620 00:17:52.919893 1859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:52.920092 1859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:52.921268 1859 hybrid_clock.cc:648] HybridClock initialized: now 1781914672921259 us; error 28 us; skew 500 ppm
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:52.923700 1859 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:17:52.924189 1859 webserver.cc:460] Webserver started at http://127.0.98.3:33965/ using document root <none> and password file <none>
I20260620 00:17:52.924412 1859 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:52.924456 1859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:52.924605 1859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:52.925680 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "2b3da69b7b3540d3bead20d2ca9279cb"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "205c9429c579784a4b5bdf4cf79b64f3"
server_key_iv: "de7496c62fab2cce741eed2b1205ef9e"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.926038 1859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "2b3da69b7b3540d3bead20d2ca9279cb"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "205c9429c579784a4b5bdf4cf79b64f3"
server_key_iv: "de7496c62fab2cce741eed2b1205ef9e"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.927403 1859 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:17:52.928192 1874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:52.928387 1859 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:52.928479 1859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "2b3da69b7b3540d3bead20d2ca9279cb"
format_stamp: "Formatted at 2026-06-20 00:17:52 on dist-test-slave-r81b"
server_key: "205c9429c579784a4b5bdf4cf79b64f3"
server_key_iv: "de7496c62fab2cce741eed2b1205ef9e"
server_key_version: "encryptionkey@0"
I20260620 00:17:52.928551 1859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:52.937156 1859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:52.938994 1859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:52.939147 1859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:52.939706 1859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:52.939765 1859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:52.939800 1859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:52.939827 1859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:52.946359 1859 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44343
I20260620 00:17:52.946424 1989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44343 every 8 connection(s)
I20260620 00:17:52.946724 1859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:52.949910 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 1859
I20260620 00:17:52.950007 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:17:52.950203 392 external_mini_cluster.cc:1428] Setting key 0a76be03ef5352606171f566ddb14ed9
I20260620 00:17:52.950354 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.940211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55433 (local address 127.0.98.62:45083)
0620 00:17:52.940286 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:52.940289 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:52.940454 (+ 165us) server_negotiation.cc:407] Connection header received
0620 00:17:52.940941 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:52.940943 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:52.940957 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:52.941045 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:52.941946 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.942795 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.944418 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.944730 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.944820 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.947608 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:52.947610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:52.947610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:52.947619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:52.948954 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.949318 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.949319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.949319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.949369 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.949592 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.949593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.949594 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.949765 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:52.949837 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:52.950171 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 00:17:52.950252 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:17:52.950639 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.940176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45083 (local address 127.0.98.3:55433)
0620 00:17:52.940316 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:52.940357 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:52.940844 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:52.941100 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:52.941104 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:52.941493 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:52.941830 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.941837 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.942901 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.942901 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.944313 (+ 1412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.944316 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.944821 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.944822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.944995 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.945687 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:52.945698 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:52.947506 (+ 1808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:52.949026 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.949031 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.949033 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.949251 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.949425 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.949426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.949426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.949518 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.949835 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:52.949841 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:52.950017 (+ 176us) client_negotiation.cc:769] Sending connection context
0620 00:17:52.950155 (+ 138us) client_negotiation.cc:240] Negotiation successful
0620 00:17:52.950326 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260620 00:17:52.951135 1990 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45083
I20260620 00:17:52.951236 1990 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:52.951403 1990 heartbeater.cc:511] Master 127.0.98.62:45083 requested a full tablet report, sending...
I20260620 00:17:52.951843 1507 ts_manager.cc:194] Registered new tserver with Master: 2b3da69b7b3540d3bead20d2ca9279cb (127.0.98.3:44343)
I20260620 00:17:52.952399 1507 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55433
I20260620 00:17:52.957091 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.952258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45541 (local address 127.0.98.62:45083)
0620 00:17:52.952299 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:52.952301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:52.952399 (+ 98us) server_negotiation.cc:407] Connection header received
0620 00:17:52.952470 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:52.952471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:52.952481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:52.952529 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:52.952935 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.953409 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.954043 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.954219 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.954267 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.954987 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:52.954988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:52.954988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:52.954994 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:52.956089 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.956548 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.956549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.956550 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.956601 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.956734 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.956734 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.956735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.956820 (+ 85us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:52.956863 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:52.956966 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:17:52.957018 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:17:52.957105 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.952200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45083 (local address 127.0.98.3:45541)
0620 00:17:52.952324 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:52.952334 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:52.952413 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:52.952600 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:52.952601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:52.952792 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:52.952858 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.952860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.953494 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.953495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.953975 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:52.953977 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.954279 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.954280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:52.954324 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.954661 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:52.954666 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:52.954918 (+ 252us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:52.956222 (+ 1304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.956223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.956223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.956483 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.956643 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:52.956643 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:52.956644 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:52.956692 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:52.956855 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:52.956856 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:52.956889 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 00:17:52.956945 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:17:52.957031 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":64,"threads_started":1}
I20260620 00:17:52.961856 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:17:52 dist-test-slave-r81b krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:52.994727 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:52.986150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35228 (local address 127.0.98.62:45083)
0620 00:17:52.986207 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:52.986209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:52.986219 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:17:52.986247 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:52.986248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:52.986258 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:52.986331 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:52.987236 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.988019 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.989417 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:52.989812 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:52.989904 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:52.992472 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:52.992474 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:52.992474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:52.992483 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:52.993749 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.994089 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.994089 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.994090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.994140 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:52.994298 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:52.994299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:52.994299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:52.994402 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:52.994475 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:52.994557 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 00:17:52.994649 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:17:52.996371 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:17:52.997958 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:17:53.004519 2011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.000395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33853 (local address 127.0.0.1:32970)
0620 00:17:53.000760 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.000772 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.000828 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.001184 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.001185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.001186 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:53.001240 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.001242 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.002484 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.002485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.003250 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.003252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.004202 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.004202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.004289 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.004305 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.004379 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.004432 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":87,"threads_started":1}
I20260620 00:17:53.004655 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.000868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35610 (local address 127.0.98.3:44343)
0620 00:17:53.001042 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.001046 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.001055 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:53.001090 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.001092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.001107 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.001183 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:53.001352 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.002124 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.003563 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.004303 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.004365 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.004431 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.004519 (+ 88us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.004555 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":72,"threads_started":1}
I20260620 00:17:53.005031 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.000503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32970 (local address 127.0.98.2:33853)
0620 00:17:53.000700 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.000705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.000897 (+ 192us) server_negotiation.cc:407] Connection header received
0620 00:17:53.001009 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.001012 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.001030 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.001115 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:53.001306 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.002379 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.003348 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.004082 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.004205 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.004295 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.004874 (+ 579us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.004920 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":92,"threads_started":1}
I20260620 00:17:53.004631 2008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.000623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44343 (local address 127.0.0.1:35610)
0620 00:17:53.000859 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.000867 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.000907 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.001241 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.001242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.001244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:53.001276 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.001277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.002209 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.002210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.003472 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.003475 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.004378 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.004379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.004444 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.004449 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.004504 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.004548 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":218,"mutex_wait_us":7,"thread_start_us":98,"threads_started":1}
I20260620 00:17:53.005301 2007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.000392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38115 (local address 127.0.0.1:38436)
0620 00:17:53.000560 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.000604 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.000672 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.001009 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.001012 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.001014 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:53.001101 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.001103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.002469 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.002470 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.003555 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.003557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.005032 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.005033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.005114 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.005119 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.005176 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.005224 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":1,"thread_start_us":62,"threads_started":1}
I20260620 00:17:53.005894 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf4a4a038014b6d945f1a88e65583ef (DEFAULT_TABLE table=test-table [id=34b96620c1474c07beab18fbc0953b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:53.005895 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf4a4a038014b6d945f1a88e65583ef (DEFAULT_TABLE table=test-table [id=34b96620c1474c07beab18fbc0953b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:53.006275 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf4a4a038014b6d945f1a88e65583ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:53.006289 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf4a4a038014b6d945f1a88e65583ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:53.008282 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.000487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.0.98.1:38115)
0620 00:17:53.000821 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.000825 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.000836 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:17:53.000877 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.000879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.000896 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.001185 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:53.001242 (+ 57us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.002380 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.004206 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.004938 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.005009 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.005090 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.005415 (+ 325us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.005465 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":65,"threads_started":1}
I20260620 00:17:53.009009 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 6cf4a4a038014b6d945f1a88e65583ef (DEFAULT_TABLE table=test-table [id=34b96620c1474c07beab18fbc0953b12]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:53.009431 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cf4a4a038014b6d945f1a88e65583ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:53.009637 2013 tablet_bootstrap.cc:492] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: Bootstrap starting.
I20260620 00:17:53.009912 2014 tablet_bootstrap.cc:492] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: Bootstrap starting.
I20260620 00:17:53.010402 2013 tablet_bootstrap.cc:654] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:53.010545 2014 tablet_bootstrap.cc:654] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:53.010767 2013 log.cc:824] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:53.010890 2014 log.cc:824] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:53.011581 2014 tablet_bootstrap.cc:492] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: No bootstrap required, opened a new log
I20260620 00:17:53.011595 2013 tablet_bootstrap.cc:492] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: No bootstrap required, opened a new log
I20260620 00:17:53.011665 2013 ts_tablet_manager.cc:1397] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:17:53.011667 2014 ts_tablet_manager.cc:1397] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:17:53.013506 2014 raft_consensus.cc:348] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.013692 2014 raft_consensus.cc:374] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:53.013741 2014 raft_consensus.cc:729] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b1413312f8428b9c1b7b0f9b61968e, State: Initialized, Role: FOLLOWER
I20260620 00:17:53.013860 2014 consensus_queue.cc:260] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.013901 2013 raft_consensus.cc:348] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.014062 2013 raft_consensus.cc:374] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:53.014096 2013 raft_consensus.cc:729] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3da69b7b3540d3bead20d2ca9279cb, State: Initialized, Role: FOLLOWER
I20260620 00:17:53.014171 2014 ts_tablet_manager.cc:1428] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 00:17:53.014249 1850 heartbeater.cc:503] Master 127.0.98.62:45083 was elected leader, sending a full tablet report...
I20260620 00:17:53.014220 2013 consensus_queue.cc:260] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.014479 2013 ts_tablet_manager.cc:1428] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:17:53.014540 1990 heartbeater.cc:503] Master 127.0.98.62:45083 was elected leader, sending a full tablet report...
I20260620 00:17:53.014693 2017 tablet_bootstrap.cc:492] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: Bootstrap starting.
I20260620 00:17:53.015336 2017 tablet_bootstrap.cc:654] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:53.015671 2017 log.cc:824] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:53.016391 2017 tablet_bootstrap.cc:492] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: No bootstrap required, opened a new log
I20260620 00:17:53.016446 2017 ts_tablet_manager.cc:1397] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:17:53.017936 2017 raft_consensus.cc:348] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.018072 2017 raft_consensus.cc:374] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:53.018096 2017 raft_consensus.cc:729] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0442db359cd3423286098ad608a4dedb, State: Initialized, Role: FOLLOWER
I20260620 00:17:53.018211 2017 consensus_queue.cc:260] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.018438 2017 ts_tablet_manager.cc:1428] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:17:53.018534 1710 heartbeater.cc:503] Master 127.0.98.62:45083 was elected leader, sending a full tablet report...
W20260620 00:17:53.041543 1851 tablet.cc:2369] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:17:53.065858 2018 raft_consensus.cc:482] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:17:53.066071 2018 raft_consensus.cc:504] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.066434 2018 leader_election.cc:288] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0442db359cd3423286098ad608a4dedb (127.0.98.1:38115), 2b3da69b7b3540d3bead20d2ca9279cb (127.0.98.3:44343)
I20260620 00:17:53.069348 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.066750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52827 (local address 127.0.98.3:44343)
0620 00:17:53.066798 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.066800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.066809 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:53.066836 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.066837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.066848 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.066926 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:53.067092 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.067583 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.068606 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.068975 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.069075 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.069086 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.069246 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.069284 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:17:53.069363 1743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.066657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44343 (local address 127.0.98.2:52827)
0620 00:17:53.066715 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.066728 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.066772 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.066930 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.066931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.066932 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:53.067015 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.067017 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.067678 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.067679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.068495 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.068497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.069080 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.069081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.069161 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.069165 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.069228 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.069272 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 00:17:53.069414 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.066740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37457 (local address 127.0.98.1:38115)
0620 00:17:53.066789 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.066791 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.066949 (+ 158us) server_negotiation.cc:407] Connection header received
0620 00:17:53.067017 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.067018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.067028 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.067094 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:53.067255 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.067716 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.068689 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.069057 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.069142 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.069154 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.069302 (+ 148us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.069340 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:17:53.069698 2022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.066678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38115 (local address 127.0.98.2:37457)
0620 00:17:53.066860 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.066877 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.066959 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.067100 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.067101 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.067103 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:53.067183 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.067184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.067802 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.067803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.068598 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.068600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.069156 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.069157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.069229 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.069232 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.069297 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.069334 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":122,"threads_started":1}
I20260620 00:17:53.069736 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf4a4a038014b6d945f1a88e65583ef" candidate_uuid: "36b1413312f8428b9c1b7b0f9b61968e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" is_pre_election: true
I20260620 00:17:53.069890 1944 raft_consensus.cc:2454] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b1413312f8428b9c1b7b0f9b61968e in term 0.
I20260620 00:17:53.070050 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf4a4a038014b6d945f1a88e65583ef" candidate_uuid: "36b1413312f8428b9c1b7b0f9b61968e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0442db359cd3423286098ad608a4dedb" is_pre_election: true
I20260620 00:17:53.070143 1736 leader_election.cc:302] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3da69b7b3540d3bead20d2ca9279cb, 36b1413312f8428b9c1b7b0f9b61968e; no voters:
I20260620 00:17:53.070186 1664 raft_consensus.cc:2454] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b1413312f8428b9c1b7b0f9b61968e in term 0.
I20260620 00:17:53.070263 2018 raft_consensus.cc:2781] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:17:53.070325 2018 raft_consensus.cc:482] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:17:53.070365 2018 raft_consensus.cc:3037] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:53.071301 2018 raft_consensus.cc:504] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.071456 2018 leader_election.cc:288] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [CANDIDATE]: Term 1 election: Requested vote from peers 0442db359cd3423286098ad608a4dedb (127.0.98.1:38115), 2b3da69b7b3540d3bead20d2ca9279cb (127.0.98.3:44343)
I20260620 00:17:53.071600 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf4a4a038014b6d945f1a88e65583ef" candidate_uuid: "36b1413312f8428b9c1b7b0f9b61968e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3da69b7b3540d3bead20d2ca9279cb"
I20260620 00:17:53.071664 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cf4a4a038014b6d945f1a88e65583ef" candidate_uuid: "36b1413312f8428b9c1b7b0f9b61968e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0442db359cd3423286098ad608a4dedb"
I20260620 00:17:53.071681 1944 raft_consensus.cc:3037] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:53.071734 1664 raft_consensus.cc:3037] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:53.072609 1944 raft_consensus.cc:2454] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b1413312f8428b9c1b7b0f9b61968e in term 1.
I20260620 00:17:53.072609 1664 raft_consensus.cc:2454] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b1413312f8428b9c1b7b0f9b61968e in term 1.
I20260620 00:17:53.072825 1735 leader_election.cc:302] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442db359cd3423286098ad608a4dedb, 36b1413312f8428b9c1b7b0f9b61968e; no voters:
I20260620 00:17:53.072970 2018 raft_consensus.cc:2781] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:53.073110 2018 raft_consensus.cc:686] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 1 LEADER]: Becoming Leader. State: Replica: 36b1413312f8428b9c1b7b0f9b61968e, State: Running, Role: LEADER
I20260620 00:17:53.073239 2018 consensus_queue.cc:237] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.074101 1507 catalog_manager.cc:5495] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e reported cstate change: term changed from 0 to 1, leader changed from <none> to 36b1413312f8428b9c1b7b0f9b61968e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "36b1413312f8428b9c1b7b0f9b61968e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } health_report { overall_health: UNKNOWN } } }
I20260620 00:17:53.099063 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.096287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.0.98.2:33853)
0620 00:17:53.096350 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.096353 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.096430 (+ 77us) server_negotiation.cc:407] Connection header received
0620 00:17:53.096595 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.096596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.096625 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.096708 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:53.096866 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.097374 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.098182 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.098379 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.098465 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.098689 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:53.098808 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:53.098893 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.098939 (+ 46us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.098963 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:17:53.101032 1944 raft_consensus.cc:1261] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Refusing update from remote peer 36b1413312f8428b9c1b7b0f9b61968e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:53.101046 1664 raft_consensus.cc:1261] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Refusing update from remote peer 36b1413312f8428b9c1b7b0f9b61968e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:53.101404 2018 consensus_queue.cc:1035] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:53.101583 2023 consensus_queue.cc:1035] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:53.104435 2026 mvcc.cc:205] Tried to move back new op lower bound from 7298722501018898432 to 7298722500908544000. Current Snapshot: MvccSnapshot[applied={T|T < 7298722501018898432}]
I20260620 00:17:53.108287 1507 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:53.108393 1507 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:53.109218 1507 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494: Sending DeleteTablet for 3 replicas of tablet 6cf4a4a038014b6d945f1a88e65583ef
I20260620 00:17:53.109550 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf4a4a038014b6d945f1a88e65583ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32970
I20260620 00:17:53.109614 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf4a4a038014b6d945f1a88e65583ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35610
I20260620 00:17:53.109789 2036 tablet_replica.cc:330] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: stopping tablet replica
I20260620 00:17:53.109891 2036 raft_consensus.cc:2229] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:17:53.109972 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cf4a4a038014b6d945f1a88e65583ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38436
I20260620 00:17:53.110030 2036 raft_consensus.cc:2258] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:53.110131 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:17:53.110297 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:17:53.110533 2036 ts_tablet_manager.cc:1905] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:53.110625 2037 tablet_replica.cc:330] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: stopping tablet replica
I20260620 00:17:53.110728 2037 raft_consensus.cc:2229] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:53.110833 2037 raft_consensus.cc:2258] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:53.111260 2037 ts_tablet_manager.cc:1905] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:53.112287 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 0844efbe548848ceb494510186800f01 (DEFAULT_TABLE table=test-table [id=4fb403d491954562b94566fce157a68b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:53.112368 2036 ts_tablet_manager.cc:1918] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:17:53.112412 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0844efbe548848ceb494510186800f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:53.112442 2036 log.cc:1196] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/6cf4a4a038014b6d945f1a88e65583ef
I20260620 00:17:53.112704 2036 ts_tablet_manager.cc:1939] T 6cf4a4a038014b6d945f1a88e65583ef P 36b1413312f8428b9c1b7b0f9b61968e: Deleting consensus metadata
I20260620 00:17:53.113091 2038 tablet_replica.cc:330] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: stopping tablet replica
I20260620 00:17:53.113222 2038 raft_consensus.cc:2229] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:53.113350 2037 ts_tablet_manager.cc:1918] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:17:53.113390 2038 raft_consensus.cc:2258] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:53.113412 2037 log.cc:1196] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/6cf4a4a038014b6d945f1a88e65583ef
I20260620 00:17:53.113649 2037 ts_tablet_manager.cc:1939] T 6cf4a4a038014b6d945f1a88e65583ef P 2b3da69b7b3540d3bead20d2ca9279cb: Deleting consensus metadata
I20260620 00:17:53.113874 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 0844efbe548848ceb494510186800f01 (DEFAULT_TABLE table=test-table [id=4fb403d491954562b94566fce157a68b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:53.114001 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0844efbe548848ceb494510186800f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:53.114050 1492 catalog_manager.cc:4841] TS 2b3da69b7b3540d3bead20d2ca9279cb (127.0.98.3:44343): tablet 6cf4a4a038014b6d945f1a88e65583ef (table test-table [id=34b96620c1474c07beab18fbc0953b12]) successfully deleted
I20260620 00:17:53.115231 2013 tablet_bootstrap.cc:492] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: Bootstrap starting.
I20260620 00:17:53.115559 2013 tablet_bootstrap.cc:654] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:53.115746 2014 tablet_bootstrap.cc:492] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: Bootstrap starting.
I20260620 00:17:53.116258 2013 tablet_bootstrap.cc:492] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: No bootstrap required, opened a new log
I20260620 00:17:53.116241 2014 tablet_bootstrap.cc:654] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:53.116292 2013 ts_tablet_manager.cc:1397] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:53.116400 2013 raft_consensus.cc:348] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.116449 2013 raft_consensus.cc:374] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:53.116463 2013 raft_consensus.cc:729] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3da69b7b3540d3bead20d2ca9279cb, State: Initialized, Role: FOLLOWER
I20260620 00:17:53.116523 2013 consensus_queue.cc:260] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.116606 2013 ts_tablet_manager.cc:1428] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:53.116940 1492 catalog_manager.cc:4841] TS 36b1413312f8428b9c1b7b0f9b61968e (127.0.98.2:33853): tablet 6cf4a4a038014b6d945f1a88e65583ef (table test-table [id=34b96620c1474c07beab18fbc0953b12]) successfully deleted
I20260620 00:17:53.117230 2014 tablet_bootstrap.cc:492] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: No bootstrap required, opened a new log
I20260620 00:17:53.117276 2014 ts_tablet_manager.cc:1397] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:17:53.117425 2014 raft_consensus.cc:348] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.117479 2014 raft_consensus.cc:374] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:53.117492 2014 raft_consensus.cc:729] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b1413312f8428b9c1b7b0f9b61968e, State: Initialized, Role: FOLLOWER
I20260620 00:17:53.117553 2014 consensus_queue.cc:260] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.117654 2014 ts_tablet_manager.cc:1428] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 00:17:53.117630 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 0844efbe548848ceb494510186800f01 (DEFAULT_TABLE table=test-table [id=4fb403d491954562b94566fce157a68b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:53.117759 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0844efbe548848ceb494510186800f01. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:53.119263 2017 tablet_bootstrap.cc:492] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: Bootstrap starting.
I20260620 00:17:53.120059 2017 tablet_bootstrap.cc:654] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:53.120796 2017 tablet_bootstrap.cc:492] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: No bootstrap required, opened a new log
I20260620 00:17:53.120863 2017 ts_tablet_manager.cc:1397] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:17:53.121025 2017 raft_consensus.cc:348] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.121111 2017 raft_consensus.cc:374] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:53.121145 2017 raft_consensus.cc:729] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0442db359cd3423286098ad608a4dedb, State: Initialized, Role: FOLLOWER
I20260620 00:17:53.121199 2017 consensus_queue.cc:260] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.121307 2017 ts_tablet_manager.cc:1428] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:17:53.121480 2038 ts_tablet_manager.cc:1905] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:53.123090 2038 ts_tablet_manager.cc:1918] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:17:53.123186 2038 log.cc:1196] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/6cf4a4a038014b6d945f1a88e65583ef
I20260620 00:17:53.123413 2038 ts_tablet_manager.cc:1939] T 6cf4a4a038014b6d945f1a88e65583ef P 0442db359cd3423286098ad608a4dedb: Deleting consensus metadata
I20260620 00:17:53.123895 1491 catalog_manager.cc:4841] TS 0442db359cd3423286098ad608a4dedb (127.0.98.1:38115): tablet 6cf4a4a038014b6d945f1a88e65583ef (table test-table [id=34b96620c1474c07beab18fbc0953b12]) successfully deleted
W20260620 00:17:53.133140 1711 tablet.cc:2369] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:17:53.181643 2023 raft_consensus.cc:482] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:17:53.181756 2023 raft_consensus.cc:504] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.181932 2023 leader_election.cc:288] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0442db359cd3423286098ad608a4dedb (127.0.98.1:38115), 2b3da69b7b3540d3bead20d2ca9279cb (127.0.98.3:44343)
I20260620 00:17:53.182217 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0844efbe548848ceb494510186800f01" candidate_uuid: "36b1413312f8428b9c1b7b0f9b61968e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0442db359cd3423286098ad608a4dedb" is_pre_election: true
I20260620 00:17:53.182201 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0844efbe548848ceb494510186800f01" candidate_uuid: "36b1413312f8428b9c1b7b0f9b61968e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" is_pre_election: true
I20260620 00:17:53.182332 1944 raft_consensus.cc:2454] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b1413312f8428b9c1b7b0f9b61968e in term 0.
I20260620 00:17:53.182330 1664 raft_consensus.cc:2454] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 36b1413312f8428b9c1b7b0f9b61968e in term 0.
I20260620 00:17:53.182507 1735 leader_election.cc:302] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442db359cd3423286098ad608a4dedb, 36b1413312f8428b9c1b7b0f9b61968e; no voters:
I20260620 00:17:53.182601 2023 raft_consensus.cc:2781] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:17:53.182646 2023 raft_consensus.cc:482] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:17:53.182682 2023 raft_consensus.cc:3037] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:53.183389 2023 raft_consensus.cc:504] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.183594 2023 leader_election.cc:288] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [CANDIDATE]: Term 1 election: Requested vote from peers 0442db359cd3423286098ad608a4dedb (127.0.98.1:38115), 2b3da69b7b3540d3bead20d2ca9279cb (127.0.98.3:44343)
I20260620 00:17:53.183759 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0844efbe548848ceb494510186800f01" candidate_uuid: "36b1413312f8428b9c1b7b0f9b61968e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0442db359cd3423286098ad608a4dedb"
I20260620 00:17:53.183854 1664 raft_consensus.cc:3037] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:53.183818 1944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0844efbe548848ceb494510186800f01" candidate_uuid: "36b1413312f8428b9c1b7b0f9b61968e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b3da69b7b3540d3bead20d2ca9279cb"
I20260620 00:17:53.183899 1944 raft_consensus.cc:3037] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:53.184790 1944 raft_consensus.cc:2454] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b1413312f8428b9c1b7b0f9b61968e in term 1.
I20260620 00:17:53.184803 1664 raft_consensus.cc:2454] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 36b1413312f8428b9c1b7b0f9b61968e in term 1.
I20260620 00:17:53.184980 1736 leader_election.cc:302] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b3da69b7b3540d3bead20d2ca9279cb, 36b1413312f8428b9c1b7b0f9b61968e; no voters:
I20260620 00:17:53.185215 2023 raft_consensus.cc:2781] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:53.185274 2023 raft_consensus.cc:686] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 1 LEADER]: Becoming Leader. State: Replica: 36b1413312f8428b9c1b7b0f9b61968e, State: Running, Role: LEADER
I20260620 00:17:53.185339 2023 consensus_queue.cc:237] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.185904 1507 catalog_manager.cc:5495] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e reported cstate change: term changed from 0 to 1, leader changed from <none> to 36b1413312f8428b9c1b7b0f9b61968e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "36b1413312f8428b9c1b7b0f9b61968e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } health_report { overall_health: UNKNOWN } } }
W20260620 00:17:53.197337 1991 tablet.cc:2369] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 00:17:53 dist-test-slave-r81b krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:53.229295 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.220943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45083 (local address 127.0.0.1:35242)
0620 00:17:53.221125 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.221139 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.221227 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.221390 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.221392 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.221844 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:53.221908 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.221912 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.222582 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.222583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.223447 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.223450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.223799 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.223800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.223872 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.224478 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:53.224484 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:53.226167 (+ 1683us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:53.227941 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:53.227949 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:53.227951 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:53.228288 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:53.228493 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:53.228493 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:53.228494 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:53.228585 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:53.228831 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:53.228839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:53.228900 (+ 61us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.228995 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.229135 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":94,"threads_started":1}
I20260620 00:17:53.229298 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.221009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35242 (local address 127.0.98.62:45083)
0620 00:17:53.221092 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.221095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.221216 (+ 121us) server_negotiation.cc:407] Connection header received
0620 00:17:53.221297 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.221298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.221308 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.221359 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:53.221995 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.222485 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.223546 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.223723 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.223770 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.226261 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:53.226263 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:53.226263 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:53.226270 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:53.227808 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:53.228365 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:53.228365 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:53.228366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:53.228420 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:53.228656 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:53.228656 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:53.228657 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:53.228762 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:53.228814 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.229047 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.229198 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":63,"server-negotiator.queue_time_us":37}
I20260620 00:17:53.232048 1507 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35242:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 00:17:53.232312 1507 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:17:53.233824 1644 tablet_service.cc:1467] Processing CreateTablet for tablet 638c5928e9934db5912a69bf6a49c414 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=20815b956375441e934dcb45dd348a7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:17:53.233851 1784 tablet_service.cc:1467] Processing CreateTablet for tablet 638c5928e9934db5912a69bf6a49c414 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=20815b956375441e934dcb45dd348a7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:17:53.233871 1924 tablet_service.cc:1467] Processing CreateTablet for tablet 638c5928e9934db5912a69bf6a49c414 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=20815b956375441e934dcb45dd348a7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:17:53.233984 1784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638c5928e9934db5912a69bf6a49c414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:53.233984 1644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638c5928e9934db5912a69bf6a49c414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:53.234020 1924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 638c5928e9934db5912a69bf6a49c414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:53.235496 2014 tablet_bootstrap.cc:492] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: Bootstrap starting.
I20260620 00:17:53.235549 2017 tablet_bootstrap.cc:492] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: Bootstrap starting.
I20260620 00:17:53.235956 2017 tablet_bootstrap.cc:654] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:53.235960 2014 tablet_bootstrap.cc:654] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:53.236402 2013 tablet_bootstrap.cc:492] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: Bootstrap starting.
I20260620 00:17:53.236779 2017 tablet_bootstrap.cc:492] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: No bootstrap required, opened a new log
I20260620 00:17:53.236835 2017 ts_tablet_manager.cc:1397] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:53.236841 2013 tablet_bootstrap.cc:654] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:53.236986 2017 raft_consensus.cc:348] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.237064 2017 raft_consensus.cc:374] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:53.237087 2017 raft_consensus.cc:729] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0442db359cd3423286098ad608a4dedb, State: Initialized, Role: FOLLOWER
I20260620 00:17:53.237129 2017 consensus_queue.cc:260] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.237231 2017 ts_tablet_manager.cc:1428] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260620 00:17:53.237347 2021 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.237416 2021 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:53.237457 2014 tablet_bootstrap.cc:492] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: No bootstrap required, opened a new log
I20260620 00:17:53.237502 2014 ts_tablet_manager.cc:1397] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:17:53.237653 2014 raft_consensus.cc:348] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.237751 2014 raft_consensus.cc:374] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:53.237771 2014 raft_consensus.cc:729] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 36b1413312f8428b9c1b7b0f9b61968e, State: Initialized, Role: FOLLOWER
I20260620 00:17:53.237819 2014 consensus_queue.cc:260] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.237932 2014 ts_tablet_manager.cc:1428] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:53.238037 2023 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.238129 2023 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:53.238590 2013 tablet_bootstrap.cc:492] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: No bootstrap required, opened a new log
I20260620 00:17:53.238633 2013 ts_tablet_manager.cc:1397] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:17:53.238795 2013 raft_consensus.cc:348] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.238857 2013 raft_consensus.cc:374] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:53.238871 2013 raft_consensus.cc:729] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b3da69b7b3540d3bead20d2ca9279cb, State: Initialized, Role: FOLLOWER
I20260620 00:17:53.238924 2013 consensus_queue.cc:260] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.239015 2013 ts_tablet_manager.cc:1428] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:53.239078 2019 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.239161 2019 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:53.276312 2019 raft_consensus.cc:482] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:17:53.276429 2019 raft_consensus.cc:504] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.276786 2019 leader_election.cc:288] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0442db359cd3423286098ad608a4dedb (127.0.98.1:38115), 36b1413312f8428b9c1b7b0f9b61968e (127.0.98.2:33853)
I20260620 00:17:53.279996 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.277056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41655 (local address 127.0.98.1:38115)
0620 00:17:53.277127 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.277129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.277138 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:53.277167 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.277168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.277176 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.277259 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:53.277422 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.278138 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.279277 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.279649 (+ 372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.279733 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.279745 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.279904 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.279943 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:17:53.280009 1883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.276952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38115 (local address 127.0.98.3:41655)
0620 00:17:53.276996 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.277008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.277054 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.277265 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.277266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.277268 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:53.277340 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.277341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.278233 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.278233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.279188 (+ 955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.279190 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.279730 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.279731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.279828 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.279832 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.279887 (+ 55us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.279930 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 00:17:53.280447 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638c5928e9934db5912a69bf6a49c414" candidate_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0442db359cd3423286098ad608a4dedb" is_pre_election: true
I20260620 00:17:53.280551 1664 raft_consensus.cc:2454] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3da69b7b3540d3bead20d2ca9279cb in term 0.
I20260620 00:17:53.280756 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.277039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60215 (local address 127.0.98.2:33853)
0620 00:17:53.277083 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.277085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.277242 (+ 157us) server_negotiation.cc:407] Connection header received
0620 00:17:53.277349 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.277350 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.277359 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.277432 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:53.277677 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.278357 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.279824 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.280406 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.280474 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.280484 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.280668 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.280711 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:17:53.280753 1875 leader_election.cc:302] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442db359cd3423286098ad608a4dedb, 2b3da69b7b3540d3bead20d2ca9279cb; no voters:
I20260620 00:17:53.280884 2019 raft_consensus.cc:2781] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:17:53.280928 2019 raft_consensus.cc:482] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:17:53.280977 2019 raft_consensus.cc:3037] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:53.281014 2052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.276952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33853 (local address 127.0.98.3:60215)
0620 00:17:53.277174 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.277181 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.277252 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.277434 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.277435 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.277436 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:53.277515 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.277516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.278453 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.278454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.279746 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.279749 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.280483 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.280484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.280579 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.280584 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.280852 (+ 268us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.280898 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":34,"thread_start_us":128,"threads_started":1}
I20260620 00:17:53.281261 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638c5928e9934db5912a69bf6a49c414" candidate_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b1413312f8428b9c1b7b0f9b61968e" is_pre_election: true
I20260620 00:17:53.281397 1804 raft_consensus.cc:2454] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b3da69b7b3540d3bead20d2ca9279cb in term 0.
I20260620 00:17:53.281840 2019 raft_consensus.cc:504] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.281986 2019 leader_election.cc:288] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [CANDIDATE]: Term 1 election: Requested vote from peers 0442db359cd3423286098ad608a4dedb (127.0.98.1:38115), 36b1413312f8428b9c1b7b0f9b61968e (127.0.98.2:33853)
I20260620 00:17:53.282137 1664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638c5928e9934db5912a69bf6a49c414" candidate_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0442db359cd3423286098ad608a4dedb"
I20260620 00:17:53.282214 1664 raft_consensus.cc:3037] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:53.282799 1664 raft_consensus.cc:2454] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3da69b7b3540d3bead20d2ca9279cb in term 1.
I20260620 00:17:53.282922 1804 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "638c5928e9934db5912a69bf6a49c414" candidate_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "36b1413312f8428b9c1b7b0f9b61968e"
I20260620 00:17:53.282976 1875 leader_election.cc:302] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0442db359cd3423286098ad608a4dedb, 2b3da69b7b3540d3bead20d2ca9279cb; no voters:
I20260620 00:17:53.282999 1804 raft_consensus.cc:3037] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:53.283098 2019 raft_consensus.cc:2781] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:53.283329 2019 raft_consensus.cc:686] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 LEADER]: Becoming Leader. State: Replica: 2b3da69b7b3540d3bead20d2ca9279cb, State: Running, Role: LEADER
I20260620 00:17:53.283421 2019 consensus_queue.cc:237] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } }
I20260620 00:17:53.283854 1804 raft_consensus.cc:2454] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b3da69b7b3540d3bead20d2ca9279cb in term 1.
I20260620 00:17:53.283833 2053 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: TxnStatusTablet state changed. Reason: New leader 2b3da69b7b3540d3bead20d2ca9279cb. Latest consensus state: current_term: 1 leader_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.284070 2053 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:17:53.284287 2055 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 00:17:53.284236 1507 catalog_manager.cc:5495] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b3da69b7b3540d3bead20d2ca9279cb (127.0.98.3). New cstate: current_term: 1 leader_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } health_report { overall_health: HEALTHY } } }
I20260620 00:17:53.284397 2055 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 00:17:53.297348 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.294801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.0.98.3:44343)
0620 00:17:53.294862 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.294865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.294874 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:53.294956 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.294957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.294975 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.295027 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:53.295179 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.295681 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.296530 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.296737 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.296814 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.296988 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:53.297109 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:53.297192 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.297246 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.297267 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:17:53.297359 2048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.294707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44343 (local address 127.0.0.1:35614)
0620 00:17:53.294785 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.294797 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.294865 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.295041 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.295041 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.295043 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:17:53.295092 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.295095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.295769 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.295770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.296433 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.296435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.296811 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.296812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.296881 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.296896 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:17:53.297176 (+ 280us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:17:53.297185 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.297241 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.297290 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:17:53.298580 1664 raft_consensus.cc:1261] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Refusing update from remote peer 2b3da69b7b3540d3bead20d2ca9279cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:53.298714 1804 raft_consensus.cc:1261] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e [term 1 FOLLOWER]: Refusing update from remote peer 2b3da69b7b3540d3bead20d2ca9279cb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:53.298862 2053 consensus_queue.cc:1035] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:53.298978 2019 consensus_queue.cc:1035] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb [LEADER]: Connected to new peer: Peer: permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:53.299716 2023 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: TxnStatusTablet state changed. Reason: New leader 2b3da69b7b3540d3bead20d2ca9279cb. Latest consensus state: current_term: 1 leader_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.299826 2023 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:53.299795 2021 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: TxnStatusTablet state changed. Reason: New leader 2b3da69b7b3540d3bead20d2ca9279cb. Latest consensus state: current_term: 1 leader_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.299916 2021 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:53.301266 2028 mvcc.cc:205] Tried to move back new op lower bound from 7298722501828816896 to 7298722501769617408. Current Snapshot: MvccSnapshot[applied={T|T < 7298722501828816896}]
I20260620 00:17:53.301349 2053 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.301442 2053 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:17:53.301496 2053 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.301539 2053 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 2b3da69b7b3540d3bead20d2ca9279cb: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:17:53.301735 2021 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.301719 2023 tablet_replica.cc:439] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 } } peers { permanent_uuid: "36b1413312f8428b9c1b7b0f9b61968e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33853 } } peers { permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 } } }
I20260620 00:17:53.301846 2023 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 36b1413312f8428b9c1b7b0f9b61968e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:53.301848 2021 tablet_replica.cc:442] T 638c5928e9934db5912a69bf6a49c414 P 0442db359cd3423286098ad608a4dedb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:53.302299 2027 mvcc.cc:205] Tried to move back new op lower bound from 7298722501828816896 to 7298722501769617408. Current Snapshot: MvccSnapshot[applied={T|T < 7298722501828816896 or (T in {7298722501828816896})}]
I20260620 00:17:53.308382 2012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.305729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42713 (local address 127.0.98.3:44343)
0620 00:17:53.305774 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.305777 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.305870 (+ 93us) server_negotiation.cc:407] Connection header received
0620 00:17:53.305945 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.305945 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.305953 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.305993 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:53.306120 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.306580 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.307688 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.307869 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.307925 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.308113 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:53.308182 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:53.308233 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.308310 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.308326 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:17:53.308444 2063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.305641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44343 (local address 127.0.98.2:42713)
0620 00:17:53.305812 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.305824 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.305887 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.306005 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.306005 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.306007 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:17:53.306066 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.306068 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.306650 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.306651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.307597 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.307600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.307944 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.307945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.308007 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.308023 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:17:53.308249 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:17:53.308251 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.308309 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.308360 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260620 00:17:53.310791 1944 raft_consensus.cc:1261] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Refusing update from remote peer 36b1413312f8428b9c1b7b0f9b61968e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:53.310945 1664 raft_consensus.cc:1261] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Refusing update from remote peer 36b1413312f8428b9c1b7b0f9b61968e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:53.311017 2031 consensus_queue.cc:1035] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b3da69b7b3540d3bead20d2ca9279cb" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:53.311116 2023 consensus_queue.cc:1035] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0442db359cd3423286098ad608a4dedb" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:53.319063 2010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.316384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47457 (local address 127.0.98.2:33853)
0620 00:17:53.316423 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.316425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.316487 (+ 62us) server_negotiation.cc:407] Connection header received
0620 00:17:53.316549 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.316550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.316563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.316616 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:53.316792 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.317294 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.318401 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.318595 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.318648 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.318832 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:53.318856 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:53.318893 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.318968 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.318993 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:17:53.319094 1996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.316342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33853 (local address 127.0.98.3:47457)
0620 00:17:53.316447 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:53.316457 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:53.316502 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:53.316637 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:53.316638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:53.316640 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:17:53.316705 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.316707 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.317369 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.317370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.318316 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:53.318319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.318668 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.318669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:53.318737 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.318754 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:17:53.318913 (+ 159us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:17:53.318915 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:17:53.318964 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 00:17:53.319010 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":66}
I20260620 00:17:53.325569 1506 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:53.325668 1506 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:53.326340 1506 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7f1af99f853c45eabf6df87418e35494: Sending DeleteTablet for 3 replicas of tablet 0844efbe548848ceb494510186800f01
I20260620 00:17:53.326627 1644 tablet_service.cc:1514] Processing DeleteTablet for tablet 0844efbe548848ceb494510186800f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38436
I20260620 00:17:53.326663 1924 tablet_service.cc:1514] Processing DeleteTablet for tablet 0844efbe548848ceb494510186800f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35610
I20260620 00:17:53.326748 2037 tablet_replica.cc:330] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: stopping tablet replica
I20260620 00:17:53.326757 2038 tablet_replica.cc:330] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: stopping tablet replica
I20260620 00:17:53.326802 2037 raft_consensus.cc:2229] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:53.326809 2038 raft_consensus.cc:2229] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:53.326840 2037 raft_consensus.cc:2258] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:53.326841 2038 raft_consensus.cc:2258] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:53.326920 1784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0844efbe548848ceb494510186800f01 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:53 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:32970
I20260620 00:17:53.327009 2036 tablet_replica.cc:330] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: stopping tablet replica
I20260620 00:17:53.327066 2036 raft_consensus.cc:2229] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:17:53.327121 2036 raft_consensus.cc:2258] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:53.327126 2038 ts_tablet_manager.cc:1905] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:53.327322 2037 ts_tablet_manager.cc:1905] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:53.328421 2036 ts_tablet_manager.cc:1905] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:53.328682 2038 ts_tablet_manager.cc:1918] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:17:53.328734 2038 log.cc:1196] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/0844efbe548848ceb494510186800f01
I20260620 00:17:53.328905 2037 ts_tablet_manager.cc:1918] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:17:53.328941 2038 ts_tablet_manager.cc:1939] T 0844efbe548848ceb494510186800f01 P 0442db359cd3423286098ad608a4dedb: Deleting consensus metadata
I20260620 00:17:53.328956 2037 log.cc:1196] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/0844efbe548848ceb494510186800f01
I20260620 00:17:53.329174 2037 ts_tablet_manager.cc:1939] T 0844efbe548848ceb494510186800f01 P 2b3da69b7b3540d3bead20d2ca9279cb: Deleting consensus metadata
I20260620 00:17:53.329249 1491 catalog_manager.cc:4841] TS 0442db359cd3423286098ad608a4dedb (127.0.98.1:38115): tablet 0844efbe548848ceb494510186800f01 (table test-table [id=4fb403d491954562b94566fce157a68b]) successfully deleted
I20260620 00:17:53.329923 1492 catalog_manager.cc:4841] TS 2b3da69b7b3540d3bead20d2ca9279cb (127.0.98.3:44343): tablet 0844efbe548848ceb494510186800f01 (table test-table [id=4fb403d491954562b94566fce157a68b]) successfully deleted
I20260620 00:17:53.330153 2036 ts_tablet_manager.cc:1918] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:17:53.330206 2036 log.cc:1196] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/0844efbe548848ceb494510186800f01
I20260620 00:17:53.330384 2036 ts_tablet_manager.cc:1939] T 0844efbe548848ceb494510186800f01 P 36b1413312f8428b9c1b7b0f9b61968e: Deleting consensus metadata
I20260620 00:17:53.330760 1492 catalog_manager.cc:4841] TS 36b1413312f8428b9c1b7b0f9b61968e (127.0.98.2:33853): tablet 0844efbe548848ceb494510186800f01 (table test-table [id=4fb403d491954562b94566fce157a68b]) successfully deleted
Jun 20 00:17:53 dist-test-slave-r81b krb5kdc[1462](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914672, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:17:53.339839 2009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.330001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.0.98.1:38115)
0620 00:17:53.330052 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.330054 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.330322 (+ 268us) server_negotiation.cc:407] Connection header received
0620 00:17:53.330423 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.330424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.330435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.330480 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:53.331255 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.331737 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.332828 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.333009 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.333067 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.336563 (+ 3496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:53.336575 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:53.336575 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:53.336584 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:53.338246 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:53.338765 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:53.338774 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:53.338774 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:53.338881 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:53.339149 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:53.339150 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:53.339150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:53.339385 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:53.339538 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.339640 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.339731 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 00:17:53.340157 1704 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38446
I20260620 00:17:53.347782 1557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:53.341130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35254 (local address 127.0.98.62:45083)
0620 00:17:53.341205 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:53.341208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:53.341325 (+ 117us) server_negotiation.cc:407] Connection header received
0620 00:17:53.341373 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:53.341374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:53.341385 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:53.341450 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:53.341907 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.342692 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.343440 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:53.343751 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:53.343825 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:53.345020 (+ 1195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:53.345022 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:53.345023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:53.345031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:53.346549 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:53.346861 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:53.346862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:53.346862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:53.346927 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:53.347099 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:53.347100 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:53.347100 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:53.347265 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:53.347366 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:53.347600 (+ 234us) server_negotiation.cc:299] Negotiation successful
0620 00:17:53.347681 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
W20260620 00:17:53.348031 1506 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35254
I20260620 00:17:53.348771 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 1579
I20260620 00:17:53.357332 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 1719
I20260620 00:17:53.363504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 1859
I20260620 00:17:53.368544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 1475
2026-06-20T00:17:53Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2443 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:17:53 dist-test-slave-r81b krb5kdc[2074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:17:53 dist-test-slave-r81b krb5kdc[2074](info): set up 2 sockets
Jun 20 00:17:53 dist-test-slave-r81b krb5kdc[2074](info): commencing operation
krb5kdc: starting...
W20260620 00:17:54.419871 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:17:54 dist-test-slave-r81b krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:17:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:17:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:17:54.539134 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40035
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40035
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:54.621562 2087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:54.621776 2087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:54.621798 2087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:54.622917 2087 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:17:54.622953 2087 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 00:17:54.622967 2087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:54.622979 2087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:54.622992 2087 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:17:54.623004 2087 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:17:54.624421 2087 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40035
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40035
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:54.624612 2087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:54.624778 2087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:54.629662 2092 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:54.629657 2095 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:54.629889 2087 server_base.cc:1034] running on GCE node
W20260620 00:17:54.629662 2093 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:54.630283 2087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:54.630638 2087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:54.631801 2087 hybrid_clock.cc:648] HybridClock initialized: now 1781914674631761 us; error 66 us; skew 500 ppm
Jun 20 00:17:54 dist-test-slave-r81b krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:54.634537 2087 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:17:54.635026 2087 webserver.cc:460] Webserver started at http://127.0.98.62:35841/ using document root <none> and password file <none>
I20260620 00:17:54.635262 2087 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:54.635306 2087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:54.635489 2087 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:54.636652 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "b92b9fd95f2e4405b9d668f29ea97318"
format_stamp: "Formatted at 2026-06-20 00:17:54 on dist-test-slave-r81b"
server_key: "9d5026b99611e45b20d9fcf2e3b03695"
server_key_iv: "1c4e3171d96aea2b6ed30522f4fd5295"
server_key_version: "encryptionkey@0"
I20260620 00:17:54.637022 2087 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "b92b9fd95f2e4405b9d668f29ea97318"
format_stamp: "Formatted at 2026-06-20 00:17:54 on dist-test-slave-r81b"
server_key: "9d5026b99611e45b20d9fcf2e3b03695"
server_key_iv: "1c4e3171d96aea2b6ed30522f4fd5295"
server_key_version: "encryptionkey@0"
I20260620 00:17:54.638379 2087 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:17:54.639146 2102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:54.639323 2087 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:17:54.639417 2087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "b92b9fd95f2e4405b9d668f29ea97318"
format_stamp: "Formatted at 2026-06-20 00:17:54 on dist-test-slave-r81b"
server_key: "9d5026b99611e45b20d9fcf2e3b03695"
server_key_iv: "1c4e3171d96aea2b6ed30522f4fd5295"
server_key_version: "encryptionkey@0"
I20260620 00:17:54.639515 2087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:54.660043 2087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:54.662531 2087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:54.662706 2087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:54.667449 2087 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40035
I20260620 00:17:54.667500 2164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40035 every 8 connection(s)
I20260620 00:17:54.667841 2087 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:17:54.668534 2165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:54.670852 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318: Bootstrap starting.
I20260620 00:17:54.671452 2165 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:54.671756 2165 log.cc:824] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:54.672467 2165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318: No bootstrap required, opened a new log
I20260620 00:17:54.674317 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 2087
I20260620 00:17:54.674434 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:17:54.674610 392 external_mini_cluster.cc:1428] Setting key b77a0c93bc3bce710af3d6d8c99a1cbf
I20260620 00:17:54.674881 2165 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92b9fd95f2e4405b9d668f29ea97318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40035 } }
I20260620 00:17:54.675055 2165 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:54.675099 2165 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92b9fd95f2e4405b9d668f29ea97318, State: Initialized, Role: FOLLOWER
I20260620 00:17:54.675261 2165 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92b9fd95f2e4405b9d668f29ea97318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40035 } }
I20260620 00:17:54.675360 2165 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:17:54.675410 2165 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:17:54.675469 2165 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:54.676257 2165 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92b9fd95f2e4405b9d668f29ea97318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40035 } }
I20260620 00:17:54.676381 2165 leader_election.cc:302] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92b9fd95f2e4405b9d668f29ea97318; no voters:
I20260620 00:17:54.676568 2165 leader_election.cc:288] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:17:54.676656 2170 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:54.676839 2165 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:17:54.676846 2170 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [term 1 LEADER]: Becoming Leader. State: Replica: b92b9fd95f2e4405b9d668f29ea97318, State: Running, Role: LEADER
I20260620 00:17:54.677104 2170 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92b9fd95f2e4405b9d668f29ea97318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40035 } }
I20260620 00:17:54.677454 2172 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92b9fd95f2e4405b9d668f29ea97318. Latest consensus state: current_term: 1 leader_uuid: "b92b9fd95f2e4405b9d668f29ea97318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92b9fd95f2e4405b9d668f29ea97318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40035 } } }
I20260620 00:17:54.677551 2172 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:54.677690 2171 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92b9fd95f2e4405b9d668f29ea97318" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92b9fd95f2e4405b9d668f29ea97318" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40035 } } }
I20260620 00:17:54.677745 2171 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318 [sys.catalog]: This master's current role is: LEADER
I20260620 00:17:54.678222 2179 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:17:54.678874 2179 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:17:54.680497 2179 catalog_manager.cc:1347] Generated new cluster ID: 2e956a01a65a43969dc5096c4d24b4ae
I20260620 00:17:54.680552 2179 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:17:54 dist-test-slave-r81b krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:54.685647 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:54.675180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46582 (local address 127.0.98.62:40035)
0620 00:17:54.675378 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:54.675390 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:54.675425 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:17:54.675607 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:54.675617 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:54.675670 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:54.675787 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:54.676649 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.678133 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.679153 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.679784 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.679923 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:54.682346 (+ 2423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:54.682350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:54.682350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:54.682362 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:54.684045 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:54.684469 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:54.684472 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:54.684473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:54.684592 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:54.684802 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:54.684803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:54.684803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:54.685041 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:54.685224 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:54.685303 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 00:17:54.685448 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":115,"threads_started":1}
I20260620 00:17:54.704943 2179 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:17:54.705619 2179 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:17:54.714417 2179 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318: Generated new TSK 0
I20260620 00:17:54.714639 2179 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:17:54.743525 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40035
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:54.827184 2191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:54.827389 2191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:54.827414 2191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:54.828632 2191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:54.828676 2191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:54.828697 2191 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 00:17:54.828709 2191 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:17:54.828769 2191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:17:54.830225 2191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:54.830499 2191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:54.830750 2191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:54.835980 2199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:54.835979 2197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:54.835968 2196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:54.836233 2191 server_base.cc:1034] running on GCE node
I20260620 00:17:54.836622 2191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:54.836901 2191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:54.838091 2191 hybrid_clock.cc:648] HybridClock initialized: now 1781914674838079 us; error 51 us; skew 500 ppm
Jun 20 00:17:54 dist-test-slave-r81b krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:54.840766 2191 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:17:54.841158 2191 webserver.cc:460] Webserver started at http://127.0.98.1:39767/ using document root <none> and password file <none>
I20260620 00:17:54.841361 2191 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:54.841403 2191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:54.841491 2191 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:54.842621 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "526c35ba0ff745a1bd3db537a4621e0c"
format_stamp: "Formatted at 2026-06-20 00:17:54 on dist-test-slave-r81b"
server_key: "7deaff7fcf96e806a270afe0bf405647"
server_key_iv: "55fc2bb7160bb563edae9512ea781283"
server_key_version: "encryptionkey@0"
I20260620 00:17:54.842934 2191 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "526c35ba0ff745a1bd3db537a4621e0c"
format_stamp: "Formatted at 2026-06-20 00:17:54 on dist-test-slave-r81b"
server_key: "7deaff7fcf96e806a270afe0bf405647"
server_key_iv: "55fc2bb7160bb563edae9512ea781283"
server_key_version: "encryptionkey@0"
I20260620 00:17:54.844322 2191 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:17:54.845052 2206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:54.845248 2191 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:17:54.845332 2191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "526c35ba0ff745a1bd3db537a4621e0c"
format_stamp: "Formatted at 2026-06-20 00:17:54 on dist-test-slave-r81b"
server_key: "7deaff7fcf96e806a270afe0bf405647"
server_key_iv: "55fc2bb7160bb563edae9512ea781283"
server_key_version: "encryptionkey@0"
I20260620 00:17:54.845388 2191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:54.854112 2191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:54.856631 2191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:54.856752 2191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:54.857226 2191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:54.857265 2191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:54.857288 2191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:54.857301 2191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:54.863821 2191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45729
I20260620 00:17:54.863899 2321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45729 every 8 connection(s)
Jun 20 00:17:54 dist-test-slave-r81b krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:54.864272 2191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:17:54.867362 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:54.857718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55333 (local address 127.0.98.62:40035)
0620 00:17:54.857795 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:54.857798 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:54.858002 (+ 204us) server_negotiation.cc:407] Connection header received
0620 00:17:54.858546 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:54.858547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:54.858559 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:54.858605 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:54.859498 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.860409 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.861490 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.861859 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.861947 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:54.864747 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:54.864748 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:54.864749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:54.864756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:54.866202 (+ 1446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:54.866580 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:54.866581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:54.866581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:54.866629 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:54.866805 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:54.866805 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:54.866806 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:54.866907 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:54.866956 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:54.867231 (+ 275us) server_negotiation.cc:299] Negotiation successful
0620 00:17:54.867298 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:17:54.867527 2215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:54.857696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40035 (local address 127.0.98.1:55333)
0620 00:17:54.857888 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:54.857929 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:54.858440 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:54.858704 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:54.858710 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:54.859082 (+ 372us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:54.859379 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:54.859385 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.860518 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.860519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:54.861417 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:54.861419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.861971 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.861971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:54.862130 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:54.862699 (+ 569us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:54.862710 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:54.864630 (+ 1920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:54.866284 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:54.866288 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:54.866290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:54.866508 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:54.866668 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:54.866669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:54.866669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:54.866756 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:54.866966 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:54.866971 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:54.867112 (+ 141us) client_negotiation.cc:769] Sending connection context
0620 00:17:54.867197 (+ 85us) client_negotiation.cc:240] Negotiation successful
0620 00:17:54.867323 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":119,"threads_started":1}
I20260620 00:17:54.868007 2322 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40035
I20260620 00:17:54.868083 2322 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:54.868294 2322 heartbeater.cc:511] Master 127.0.98.62:40035 requested a full tablet report, sending...
I20260620 00:17:54.868600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 2191
I20260620 00:17:54.868700 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:17:54.868880 392 external_mini_cluster.cc:1428] Setting key 57c0d555e5bcc22c885a85ca956a7c6d
I20260620 00:17:54.868880 2118 ts_manager.cc:194] Registered new tserver with Master: 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729)
I20260620 00:17:54.869727 2118 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55333
I20260620 00:17:54.875790 2328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:54.869114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40035 (local address 127.0.98.1:32771)
0620 00:17:54.869251 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:54.869261 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:54.869317 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:54.870149 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:54.870150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:54.870370 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:54.870447 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:54.870450 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.871114 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.871115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:54.871631 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:54.871633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.872046 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.872047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:54.872120 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:54.872633 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:54.872637 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:54.872898 (+ 261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:54.874851 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:54.874852 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:54.874852 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:54.875016 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:54.875231 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:54.875232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:54.875232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:54.875283 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:54.875545 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:54.875546 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:54.875586 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 00:17:54.875636 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:17:54.875715 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
I20260620 00:17:54.875857 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:54.869934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32771 (local address 127.0.98.62:40035)
0620 00:17:54.870013 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:54.870015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:54.870022 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:17:54.870040 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:54.870041 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:54.870050 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:54.870111 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:54.870538 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.871005 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.871717 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:54.871959 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:54.872026 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:54.872987 (+ 961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:54.872988 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:54.872989 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:54.872996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:54.874768 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:54.875097 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:54.875099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:54.875099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:54.875162 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:54.875341 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:54.875342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:54.875343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:54.875480 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:54.875552 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:54.875663 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 00:17:54.875744 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:17:54.896116 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40035
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:54.980679 2331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:54.980901 2331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:54.980926 2331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:54.982120 2331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:54.982164 2331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:54.982187 2331 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 00:17:54.982241 2331 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:17:54.982267 2331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:17:54.983793 2331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2331
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:54.984063 2331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:54.984333 2331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:54.989656 2339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:54.989660 2337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:54.989696 2336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:54.989884 2331 server_base.cc:1034] running on GCE node
I20260620 00:17:54.990116 2331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:54.990401 2331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:54.991572 2331 hybrid_clock.cc:648] HybridClock initialized: now 1781914674991543 us; error 37 us; skew 500 ppm
Jun 20 00:17:54 dist-test-slave-r81b krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:54.994328 2331 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:17:54.994809 2331 webserver.cc:460] Webserver started at http://127.0.98.2:36807/ using document root <none> and password file <none>
I20260620 00:17:54.995059 2331 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:54.995107 2331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:54.995241 2331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:54.996436 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "12a9ca5e181b4eeebd1ee4ba5882589c"
format_stamp: "Formatted at 2026-06-20 00:17:54 on dist-test-slave-r81b"
server_key: "5fd1d840fcb0ad977cb802a7860df24c"
server_key_iv: "4105edbcf556f2c352d42dea23642b9b"
server_key_version: "encryptionkey@0"
I20260620 00:17:54.996798 2331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "12a9ca5e181b4eeebd1ee4ba5882589c"
format_stamp: "Formatted at 2026-06-20 00:17:54 on dist-test-slave-r81b"
server_key: "5fd1d840fcb0ad977cb802a7860df24c"
server_key_iv: "4105edbcf556f2c352d42dea23642b9b"
server_key_version: "encryptionkey@0"
I20260620 00:17:54.998135 2331 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:17:54.998935 2346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:54.999161 2331 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:54.999235 2331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "12a9ca5e181b4eeebd1ee4ba5882589c"
format_stamp: "Formatted at 2026-06-20 00:17:54 on dist-test-slave-r81b"
server_key: "5fd1d840fcb0ad977cb802a7860df24c"
server_key_iv: "4105edbcf556f2c352d42dea23642b9b"
server_key_version: "encryptionkey@0"
I20260620 00:17:54.999325 2331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:55.013130 2331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:55.015619 2331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:55.015774 2331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:55.016436 2331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:55.016477 2331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:55.016501 2331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:55.016513 2331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:55.023630 2331 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33873
I20260620 00:17:55.023680 2461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33873 every 8 connection(s)
I20260620 00:17:55.023993 2331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:17:55 dist-test-slave-r81b krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:55.027628 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.016971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49837 (local address 127.0.98.62:40035)
0620 00:17:55.017047 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.017049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.017324 (+ 275us) server_negotiation.cc:407] Connection header received
0620 00:17:55.017898 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.017900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.017914 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.018003 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:55.018966 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.019798 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.021553 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.021757 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.021817 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.024659 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:55.024660 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:55.024661 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:55.024669 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:55.026188 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.026629 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.026630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.026631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.026700 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.026960 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.026961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.026961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.027082 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:55.027155 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.027442 (+ 287us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.027525 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:17:55.027817 2355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.016960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40035 (local address 127.0.98.2:49837)
0620 00:17:55.017182 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.017225 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.017778 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.018094 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.018099 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.018514 (+ 415us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:55.018816 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.018821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.019919 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.019920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.021456 (+ 1536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.021460 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.021850 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.021851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.022010 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.022670 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:55.022681 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:55.024542 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:55.026291 (+ 1749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.026298 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.026300 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.026534 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.026769 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.026770 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.026770 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.026889 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.027160 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:55.027166 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:55.027309 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.027420 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.027584 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":147,"threads_started":1}
I20260620 00:17:55.028368 2462 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40035
I20260620 00:17:55.028471 2462 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:55.028702 2462 heartbeater.cc:511] Master 127.0.98.62:40035 requested a full tablet report, sending...
I20260620 00:17:55.029088 2119 ts_manager.cc:194] Registered new tserver with Master: 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873)
I20260620 00:17:55.029474 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49837
I20260620 00:17:55.032549 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 2331
I20260620 00:17:55.032625 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:17:55.032819 392 external_mini_cluster.cc:1428] Setting key 75fbf26ad69a87bd5692288dac27d866
I20260620 00:17:55.035414 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.029545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45399 (local address 127.0.98.62:40035)
0620 00:17:55.029585 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.029587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.029697 (+ 110us) server_negotiation.cc:407] Connection header received
0620 00:17:55.029752 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.029753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.029761 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.029811 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:55.030157 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.030620 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.031302 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.031492 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.031581 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.032499 (+ 918us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:55.032500 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:55.032500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:55.032508 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:55.034247 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.034579 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.034580 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.034581 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.034640 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.034866 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.034866 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.034867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.035067 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:55.035121 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.035296 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.035355 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:17:55.035524 2468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.029489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40035 (local address 127.0.98.2:45399)
0620 00:17:55.029640 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.029651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.029715 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.029858 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.029860 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.030012 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:55.030068 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.030069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.030709 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.030710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.031218 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.031219 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.031582 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.031583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.031665 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.032045 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:55.032049 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:55.032415 (+ 366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:55.034333 (+ 1918us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.034334 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.034335 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.034501 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.034704 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.034705 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.034705 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.034792 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.035134 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:55.035135 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:55.035188 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.035268 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.035387 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:17:55.058916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40035
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:17:55.142817 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:17:55.143041 2471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:17:55.143067 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:17:55.144349 2471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:17:55.144399 2471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:17:55.144421 2471 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260620 00:17:55.144433 2471 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:17:55.144479 2471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:17:55.146016 2471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:40035
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:17:55.146278 2471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:17:55.146523 2471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:17:55.152642 2477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:55.152652 2479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:17:55.152642 2476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:17:55.152896 2471 server_base.cc:1034] running on GCE node
I20260620 00:17:55.153185 2471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:17:55.153452 2471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:17:55.154604 2471 hybrid_clock.cc:648] HybridClock initialized: now 1781914675154592 us; error 28 us; skew 500 ppm
Jun 20 00:17:55 dist-test-slave-r81b krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:17:55.157317 2471 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:17:55.157783 2471 webserver.cc:460] Webserver started at http://127.0.98.3:38597/ using document root <none> and password file <none>
I20260620 00:17:55.157974 2471 fs_manager.cc:359] Metadata directory not provided
I20260620 00:17:55.158015 2471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:17:55.158113 2471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:17:55.159142 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "90d1c965abc3415fb35bc179549f0e6f"
format_stamp: "Formatted at 2026-06-20 00:17:55 on dist-test-slave-r81b"
server_key: "78d5b28e3549cae1d2de2745e276b7aa"
server_key_iv: "e20f13f828058c19d97d36b7b4d52453"
server_key_version: "encryptionkey@0"
I20260620 00:17:55.159448 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "90d1c965abc3415fb35bc179549f0e6f"
format_stamp: "Formatted at 2026-06-20 00:17:55 on dist-test-slave-r81b"
server_key: "78d5b28e3549cae1d2de2745e276b7aa"
server_key_iv: "e20f13f828058c19d97d36b7b4d52453"
server_key_version: "encryptionkey@0"
I20260620 00:17:55.160962 2471 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:17:55.161744 2486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:55.162031 2471 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:55.162164 2471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "90d1c965abc3415fb35bc179549f0e6f"
format_stamp: "Formatted at 2026-06-20 00:17:55 on dist-test-slave-r81b"
server_key: "78d5b28e3549cae1d2de2745e276b7aa"
server_key_iv: "e20f13f828058c19d97d36b7b4d52453"
server_key_version: "encryptionkey@0"
I20260620 00:17:55.162256 2471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:17:55.179610 2471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:17:55.181792 2471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:17:55.181957 2471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:17:55.182454 2471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:17:55.182523 2471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:55.182565 2471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:17:55.182592 2471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:17:55 dist-test-slave-r81b krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:55.189890 2471 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38445
I20260620 00:17:55.190025 2601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38445 every 8 connection(s)
I20260620 00:17:55.190290 2471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:17:55.193517 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.183003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37453 (local address 127.0.98.62:40035)
0620 00:17:55.183077 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.183080 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.183324 (+ 244us) server_negotiation.cc:407] Connection header received
0620 00:17:55.183842 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.183843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.183856 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.183905 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:55.184759 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.185586 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.187108 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.187435 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.187507 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.190288 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:55.190290 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:55.190290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:55.190299 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:55.191857 (+ 1558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.192362 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.192363 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.192363 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.192429 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.192691 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.192692 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.192692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.192814 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:55.192893 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.193331 (+ 438us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.193421 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:17:55.193763 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.182976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40035 (local address 127.0.98.3:37453)
0620 00:17:55.183172 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.183212 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.183713 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.183981 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.183985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.184376 (+ 391us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:55.184672 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.184679 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.185680 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.185681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.187016 (+ 1335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.187019 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.187548 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.187549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.187703 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.188303 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:55.188318 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:55.190173 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:55.191959 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.191966 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.191968 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.192274 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.192496 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.192497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.192497 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.192616 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.192885 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:55.192891 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:55.193129 (+ 238us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.193288 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.193477 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":126,"threads_started":1}
I20260620 00:17:55.194116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 2471
I20260620 00:17:55.194216 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:17:55.194267 2602 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40035
I20260620 00:17:55.194345 2602 heartbeater.cc:464] Registering TS with master...
I20260620 00:17:55.194408 392 external_mini_cluster.cc:1428] Setting key 52ff98a41f63e0cbf8f40d6fc85c9d80
I20260620 00:17:55.194545 2602 heartbeater.cc:511] Master 127.0.98.62:40035 requested a full tablet report, sending...
I20260620 00:17:55.195434 2119 ts_manager.cc:194] Registered new tserver with Master: 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3:38445)
I20260620 00:17:55.195870 2119 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37453
I20260620 00:17:55.201574 2608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.195389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40035 (local address 127.0.98.3:60183)
0620 00:17:55.195528 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.195540 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.195609 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.196120 (+ 511us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.196121 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.196370 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:55.196431 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.196433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.197060 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.197061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.197542 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.197543 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.198083 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.198084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.198143 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.198742 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:55.198746 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:55.199057 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:55.200696 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.200697 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.200697 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.200867 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.201050 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.201051 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.201051 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.201105 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.201317 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:55.201318 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:55.201355 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.201406 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.201486 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260620 00:17:55.201591 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.195415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60183 (local address 127.0.98.62:40035)
0620 00:17:55.196003 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.196013 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.196021 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:17:55.196040 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.196040 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.196050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.196105 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:55.196517 (+ 412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.196966 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.197804 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.198002 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.198067 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.199156 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:55.199157 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:55.199157 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:55.199165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:55.200620 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.200937 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.200938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.200938 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.200992 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.201159 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.201160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.201160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.201263 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:55.201318 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.201460 (+ 142us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.201525 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569}
I20260620 00:17:55.206223 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:17:55 dist-test-slave-r81b krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:17:55 dist-test-slave-r81b krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914675, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:55.230080 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.222259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.0.98.62:40035)
0620 00:17:55.222327 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.222329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.222338 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:55.222398 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.222399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.222409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.222460 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:55.223229 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.223734 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.224600 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.224968 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.225059 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.227486 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:55.227488 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:55.227488 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:55.227495 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:55.228997 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.229325 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.229326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.229326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.229451 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.229630 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.229631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.229631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.229741 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:55.229817 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.229918 (+ 101us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.229979 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:17:55.231868 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:17:55.233635 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:17:55.240386 2619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.236068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38445 (local address 127.0.0.1:45768)
0620 00:17:55.236258 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.236298 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.236366 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.236743 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.236745 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.236747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:55.236827 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.236829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.237917 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.237918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.239270 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.239272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.240130 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.240156 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.240209 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.240212 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.240259 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.240293 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":56,"threads_started":1}
I20260620 00:17:55.240432 2624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.236440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45768 (local address 127.0.98.3:38445)
0620 00:17:55.236597 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.236602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.236612 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:17:55.236650 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.236653 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.236669 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.236742 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:55.237035 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.237802 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.239347 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.240022 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.240115 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.240188 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.240283 (+ 95us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.240331 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":81,"threads_started":1}
I20260620 00:17:55.240535 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.236353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48160 (local address 127.0.98.1:45729)
0620 00:17:55.236546 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.236551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.236725 (+ 174us) server_negotiation.cc:407] Connection header received
0620 00:17:55.236766 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.236768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.236781 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.236853 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:55.236963 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.238140 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.239394 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.239870 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.239964 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.240030 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.240359 (+ 329us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.240417 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":113,"threads_started":1}
I20260620 00:17:55.240319 2621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.236081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45729 (local address 127.0.0.1:48160)
0620 00:17:55.236597 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.236608 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.236668 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.236849 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.236850 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.236852 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:55.236893 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.236895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.238228 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.238229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.239331 (+ 1102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.239333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.239956 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.239956 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.240042 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.240056 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.240129 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.240225 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"mutex_wait_us":29,"thread_start_us":114,"threads_started":1}
I20260620 00:17:55.241012 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.236277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.0.98.2:33873)
0620 00:17:55.236417 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.236422 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.236608 (+ 186us) server_negotiation.cc:407] Connection header received
0620 00:17:55.236683 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.236686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.236703 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.236780 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:55.237072 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.238140 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.239227 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.239955 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.240028 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.240106 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.240333 (+ 227us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.240374 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260620 00:17:55.241212 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9cabd75a1c407394f17dd6bd8c9c48 (DEFAULT_TABLE table=test-table [id=5a603ea3c873493aaa34fc63f5323519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:55.241328 2529 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9cabd75a1c407394f17dd6bd8c9c48 (DEFAULT_TABLE table=test-table [id=5a603ea3c873493aaa34fc63f5323519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:55.241535 2620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.236068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.0.1:41298)
0620 00:17:55.236546 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.236558 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.236613 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.236912 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.236913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.236916 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:55.236978 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.236979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.238244 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.238245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.239139 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.239142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.240171 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.240172 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.240235 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.240239 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.241331 (+ 1092us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.241439 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":5,"thread_start_us":97,"threads_started":1}
I20260620 00:17:55.241641 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9cabd75a1c407394f17dd6bd8c9c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:55.241698 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9cabd75a1c407394f17dd6bd8c9c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:55.242137 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 7b9cabd75a1c407394f17dd6bd8c9c48 (DEFAULT_TABLE table=test-table [id=5a603ea3c873493aaa34fc63f5323519]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:55.242436 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b9cabd75a1c407394f17dd6bd8c9c48. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:55.244887 2625 tablet_bootstrap.cc:492] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: Bootstrap starting.
I20260620 00:17:55.245236 2626 tablet_bootstrap.cc:492] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: Bootstrap starting.
I20260620 00:17:55.245544 2625 tablet_bootstrap.cc:654] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:55.245848 2625 log.cc:824] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:55.246174 2627 tablet_bootstrap.cc:492] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: Bootstrap starting.
I20260620 00:17:55.246212 2626 tablet_bootstrap.cc:654] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:55.246472 2626 log.cc:824] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:55.246757 2627 tablet_bootstrap.cc:654] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:55.247032 2627 log.cc:824] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: Log is configured to *not* fsync() on all Append() calls
I20260620 00:17:55.247371 2626 tablet_bootstrap.cc:492] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: No bootstrap required, opened a new log
I20260620 00:17:55.247448 2626 ts_tablet_manager.cc:1397] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:17:55.247881 2625 tablet_bootstrap.cc:492] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: No bootstrap required, opened a new log
I20260620 00:17:55.247997 2625 ts_tablet_manager.cc:1397] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:17:55.248157 2627 tablet_bootstrap.cc:492] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: No bootstrap required, opened a new log
I20260620 00:17:55.248219 2627 ts_tablet_manager.cc:1397] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:17:55.249204 2626 raft_consensus.cc:348] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } }
I20260620 00:17:55.249359 2626 raft_consensus.cc:374] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:55.249401 2626 raft_consensus.cc:729] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d1c965abc3415fb35bc179549f0e6f, State: Initialized, Role: FOLLOWER
I20260620 00:17:55.249544 2626 consensus_queue.cc:260] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } }
I20260620 00:17:55.249831 2626 ts_tablet_manager.cc:1428] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:17:55.249832 2602 heartbeater.cc:503] Master 127.0.98.62:40035 was elected leader, sending a full tablet report...
I20260620 00:17:55.250104 2625 raft_consensus.cc:348] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } }
I20260620 00:17:55.250174 2627 raft_consensus.cc:348] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } }
I20260620 00:17:55.250336 2625 raft_consensus.cc:374] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:55.250335 2627 raft_consensus.cc:374] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:55.250383 2627 raft_consensus.cc:729] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a9ca5e181b4eeebd1ee4ba5882589c, State: Initialized, Role: FOLLOWER
I20260620 00:17:55.250383 2625 raft_consensus.cc:729] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c35ba0ff745a1bd3db537a4621e0c, State: Initialized, Role: FOLLOWER
I20260620 00:17:55.250502 2625 consensus_queue.cc:260] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } }
I20260620 00:17:55.250547 2627 consensus_queue.cc:260] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } }
I20260620 00:17:55.250790 2625 ts_tablet_manager.cc:1428] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:17:55.250790 2627 ts_tablet_manager.cc:1428] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:17:55.250926 2322 heartbeater.cc:503] Master 127.0.98.62:40035 was elected leader, sending a full tablet report...
I20260620 00:17:55.251222 2462 heartbeater.cc:503] Master 127.0.98.62:40035 was elected leader, sending a full tablet report...
W20260620 00:17:55.274677 2463 tablet.cc:2369] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:17:55.365094 2323 tablet.cc:2369] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:17:55.440879 2603 tablet.cc:2369] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:17:55.575702 2631 raft_consensus.cc:482] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:17:55.575834 2631 raft_consensus.cc:504] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } }
I20260620 00:17:55.576256 2631 leader_election.cc:288] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873), 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729)
I20260620 00:17:55.580253 2495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.576460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.98.3:46919)
0620 00:17:55.576546 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.576560 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.576613 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.576793 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.576794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.576797 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:55.576872 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.576874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.577881 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.577882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.579109 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.579112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.579948 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.579949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.580030 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.580034 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.580129 (+ 95us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.580183 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260620 00:17:55.580268 2634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.576582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45729 (local address 127.0.98.3:57295)
0620 00:17:55.576783 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.576793 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.576850 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.577036 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.577037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.577039 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:55.577114 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.577116 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.578036 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.578036 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.579109 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.579112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.579949 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.579949 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.580030 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.580034 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.580102 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.580163 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":9,"thread_start_us":118,"threads_started":1}
I20260620 00:17:55.580293 2622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.576579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46919 (local address 127.0.98.2:33873)
0620 00:17:55.576631 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.576634 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.576667 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:17:55.576700 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.576701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.576716 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.576781 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:55.576954 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.577804 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.579209 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.579861 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.579944 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.579955 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.580171 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.580216 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:17:55.580293 2623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.576641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57295 (local address 127.0.98.1:45729)
0620 00:17:55.576683 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.576685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.576856 (+ 171us) server_negotiation.cc:407] Connection header received
0620 00:17:55.576892 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.576893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.576904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.577012 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:55.577185 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.577973 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.579210 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.579829 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.579913 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.579927 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.580145 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.580214 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:17:55.580626 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9cabd75a1c407394f17dd6bd8c9c48" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" is_pre_election: true
I20260620 00:17:55.580791 2416 raft_consensus.cc:2454] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 0.
I20260620 00:17:55.580751 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9cabd75a1c407394f17dd6bd8c9c48" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c35ba0ff745a1bd3db537a4621e0c" is_pre_election: true
I20260620 00:17:55.580921 2276 raft_consensus.cc:2454] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 0.
I20260620 00:17:55.581048 2489 leader_election.cc:302] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a9ca5e181b4eeebd1ee4ba5882589c, 90d1c965abc3415fb35bc179549f0e6f; no voters:
I20260620 00:17:55.581187 2631 raft_consensus.cc:2781] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:17:55.581240 2631 raft_consensus.cc:482] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:17:55.581295 2631 raft_consensus.cc:3037] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:55.582284 2631 raft_consensus.cc:504] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } }
I20260620 00:17:55.582427 2631 leader_election.cc:288] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 election: Requested vote from peers 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873), 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729)
I20260620 00:17:55.582563 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9cabd75a1c407394f17dd6bd8c9c48" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c"
I20260620 00:17:55.582649 2416 raft_consensus.cc:3037] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:55.582656 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b9cabd75a1c407394f17dd6bd8c9c48" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c35ba0ff745a1bd3db537a4621e0c"
I20260620 00:17:55.582746 2276 raft_consensus.cc:3037] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:55.583357 2416 raft_consensus.cc:2454] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 1.
I20260620 00:17:55.583530 2489 leader_election.cc:302] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a9ca5e181b4eeebd1ee4ba5882589c, 90d1c965abc3415fb35bc179549f0e6f; no voters:
I20260620 00:17:55.583567 2276 raft_consensus.cc:2454] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 1.
I20260620 00:17:55.583614 2631 raft_consensus.cc:2781] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:55.583752 2631 raft_consensus.cc:686] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 1 LEADER]: Becoming Leader. State: Replica: 90d1c965abc3415fb35bc179549f0e6f, State: Running, Role: LEADER
I20260620 00:17:55.583861 2631 consensus_queue.cc:237] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } }
I20260620 00:17:55.584703 2118 catalog_manager.cc:5495] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } health_report { overall_health: UNKNOWN } } }
Jun 20 00:17:55 dist-test-slave-r81b krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:55.612498 2169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.605009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46604 (local address 127.0.98.62:40035)
0620 00:17:55.605092 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.605095 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.605199 (+ 104us) server_negotiation.cc:407] Connection header received
0620 00:17:55.605230 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.605231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.605241 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.605309 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:55.605758 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.606422 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.607134 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.607329 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.607388 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.609739 (+ 2351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:55.609745 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:55.609745 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:55.609752 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:55.611182 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.611670 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.611671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.611672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.611727 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:55.611928 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:55.611928 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:55.611929 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:55.612039 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:55.612111 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.612283 (+ 172us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.612384 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 00:17:55.612576 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.604931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40035 (local address 127.0.0.1:46604)
0620 00:17:55.605105 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.605115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.605183 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.605323 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.605323 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.605640 (+ 317us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:17:55.605689 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.605691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.606503 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.606504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.607060 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.607061 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.607395 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.607396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.607438 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.608117 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:17:55.608122 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:17:55.609637 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:17:55.611292 (+ 1655us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.611300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.611303 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.611579 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.611793 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:17:55.611794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:17:55.611795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:17:55.611861 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:17:55.612108 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:17:55.612116 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:17:55.612188 (+ 72us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.612261 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.612449 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":78,"thread_start_us":73,"threads_started":1}
I20260620 00:17:55.614614 2118 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46604:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 00:17:55.614833 2118 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:17:55.616420 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 5075047a07124bf7b6d0280b60223423 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28eb790e8aaf402283f2b5e0cbf74c81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:17:55.616420 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 5075047a07124bf7b6d0280b60223423 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28eb790e8aaf402283f2b5e0cbf74c81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:17:55.616420 2529 tablet_service.cc:1467] Processing CreateTablet for tablet 5075047a07124bf7b6d0280b60223423 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=28eb790e8aaf402283f2b5e0cbf74c81]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:17:55.616590 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075047a07124bf7b6d0280b60223423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:55.616588 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075047a07124bf7b6d0280b60223423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:55.616588 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5075047a07124bf7b6d0280b60223423. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:55.617715 2625 tablet_bootstrap.cc:492] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: Bootstrap starting.
I20260620 00:17:55.618074 2627 tablet_bootstrap.cc:492] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: Bootstrap starting.
I20260620 00:17:55.618083 2626 tablet_bootstrap.cc:492] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: Bootstrap starting.
I20260620 00:17:55.618127 2625 tablet_bootstrap.cc:654] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:55.618459 2627 tablet_bootstrap.cc:654] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:55.618628 2626 tablet_bootstrap.cc:654] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:55.618857 2625 tablet_bootstrap.cc:492] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: No bootstrap required, opened a new log
I20260620 00:17:55.618903 2625 ts_tablet_manager.cc:1397] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:55.619051 2625 raft_consensus.cc:348] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.619133 2625 raft_consensus.cc:374] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:55.619163 2625 raft_consensus.cc:729] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c35ba0ff745a1bd3db537a4621e0c, State: Initialized, Role: FOLLOWER
I20260620 00:17:55.619205 2625 consensus_queue.cc:260] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.619306 2626 tablet_bootstrap.cc:492] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: No bootstrap required, opened a new log
I20260620 00:17:55.619325 2625 ts_tablet_manager.cc:1428] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:55.619364 2626 ts_tablet_manager.cc:1397] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:55.619428 2632 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.619527 2632 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:55.619505 2626 raft_consensus.cc:348] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.619589 2626 raft_consensus.cc:374] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:55.619621 2626 raft_consensus.cc:729] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d1c965abc3415fb35bc179549f0e6f, State: Initialized, Role: FOLLOWER
I20260620 00:17:55.619668 2626 consensus_queue.cc:260] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.619771 2626 ts_tablet_manager.cc:1428] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:17:55.619896 2631 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.620018 2631 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:55.620240 2627 tablet_bootstrap.cc:492] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: No bootstrap required, opened a new log
I20260620 00:17:55.620294 2627 ts_tablet_manager.cc:1397] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:17:55.620461 2627 raft_consensus.cc:348] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.620533 2627 raft_consensus.cc:374] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:55.620568 2627 raft_consensus.cc:729] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a9ca5e181b4eeebd1ee4ba5882589c, State: Initialized, Role: FOLLOWER
I20260620 00:17:55.620616 2627 consensus_queue.cc:260] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.620731 2627 ts_tablet_manager.cc:1428] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:55.620802 2633 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.620879 2633 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:55.845481 2631 raft_consensus.cc:482] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:17:55.845609 2631 raft_consensus.cc:504] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.845793 2631 leader_election.cc:288] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729), 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873)
I20260620 00:17:55.845992 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075047a07124bf7b6d0280b60223423" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c35ba0ff745a1bd3db537a4621e0c" is_pre_election: true
I20260620 00:17:55.846009 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075047a07124bf7b6d0280b60223423" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" is_pre_election: true
I20260620 00:17:55.846118 2276 raft_consensus.cc:2454] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 0.
I20260620 00:17:55.846123 2416 raft_consensus.cc:2454] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 0.
I20260620 00:17:55.846290 2490 leader_election.cc:302] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526c35ba0ff745a1bd3db537a4621e0c, 90d1c965abc3415fb35bc179549f0e6f; no voters:
I20260620 00:17:55.846412 2631 raft_consensus.cc:2781] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:17:55.846473 2631 raft_consensus.cc:482] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:17:55.846490 2631 raft_consensus.cc:3037] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:55.847182 2631 raft_consensus.cc:504] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.847337 2631 leader_election.cc:288] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 election: Requested vote from peers 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729), 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873)
I20260620 00:17:55.847451 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075047a07124bf7b6d0280b60223423" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c35ba0ff745a1bd3db537a4621e0c"
I20260620 00:17:55.847525 2276 raft_consensus.cc:3037] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:55.847584 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5075047a07124bf7b6d0280b60223423" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c"
I20260620 00:17:55.847680 2416 raft_consensus.cc:3037] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:55.848243 2276 raft_consensus.cc:2454] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 1.
I20260620 00:17:55.848405 2490 leader_election.cc:302] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526c35ba0ff745a1bd3db537a4621e0c, 90d1c965abc3415fb35bc179549f0e6f; no voters:
I20260620 00:17:55.848445 2416 raft_consensus.cc:2454] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 1.
I20260620 00:17:55.848510 2631 raft_consensus.cc:2781] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:55.848563 2631 raft_consensus.cc:686] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [term 1 LEADER]: Becoming Leader. State: Replica: 90d1c965abc3415fb35bc179549f0e6f, State: Running, Role: LEADER
I20260620 00:17:55.848616 2631 consensus_queue.cc:237] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.848891 2635 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: TxnStatusTablet state changed. Reason: New leader 90d1c965abc3415fb35bc179549f0e6f. Latest consensus state: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.849012 2635 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:17:55.849201 2647 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 00:17:55.849347 2647 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 00:17:55.849293 2118 catalog_manager.cc:5495] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } health_report { overall_health: HEALTHY } } }
I20260620 00:17:55.907868 2642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.904411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38445 (local address 127.0.0.1:45784)
0620 00:17:55.904511 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.904530 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.904580 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.904971 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.904972 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.904974 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:17:55.905023 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.905025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.905956 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.905957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.906705 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.906708 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.907203 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.907204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.907259 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.907276 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:17:55.907686 (+ 410us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:17:55.907697 (+ 11us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.907755 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.907804 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260620 00:17:55.907891 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.904668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45784 (local address 127.0.98.3:38445)
0620 00:17:55.904794 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.904796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.904806 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:17:55.904843 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.904844 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.904865 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.904951 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:55.905097 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.905857 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.906785 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.907116 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.907189 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.907383 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:55.907584 (+ 201us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:55.907673 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.907759 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.907783 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":56,"threads_started":1}
I20260620 00:17:55.909945 2416 raft_consensus.cc:1261] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Refusing update from remote peer 90d1c965abc3415fb35bc179549f0e6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:55.909965 2276 raft_consensus.cc:1261] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Refusing update from remote peer 90d1c965abc3415fb35bc179549f0e6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:55.910318 2631 consensus_queue.cc:1035] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:55.910466 2635 consensus_queue.cc:1035] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:55.911684 2633 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: TxnStatusTablet state changed. Reason: New leader 90d1c965abc3415fb35bc179549f0e6f. Latest consensus state: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.911697 2632 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: TxnStatusTablet state changed. Reason: New leader 90d1c965abc3415fb35bc179549f0e6f. Latest consensus state: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.911779 2633 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:55.911798 2632 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:55.913163 2635 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.913254 2635 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:17:55.913350 2635 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.913326 2632 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.913403 2635 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 90d1c965abc3415fb35bc179549f0e6f: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:17:55.913414 2632 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 526c35ba0ff745a1bd3db537a4621e0c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:55.913879 2651 mvcc.cc:205] Tried to move back new op lower bound from 7298722512524058624 to 7298722512276267008. Current Snapshot: MvccSnapshot[applied={T|T < 7298722512524058624}]
I20260620 00:17:55.913949 2633 tablet_replica.cc:439] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } } }
I20260620 00:17:55.914050 2633 tablet_replica.cc:442] T 5075047a07124bf7b6d0280b60223423 P 12a9ca5e181b4eeebd1ee4ba5882589c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:17:55.914300 2653 mvcc.cc:205] Tried to move back new op lower bound from 7298722512524058624 to 7298722512276267008. Current Snapshot: MvccSnapshot[applied={T|T < 7298722512524058624}]
I20260620 00:17:55.914184 2652 mvcc.cc:205] Tried to move back new op lower bound from 7298722512524058624 to 7298722512276267008. Current Snapshot: MvccSnapshot[applied={T|T < 7298722512524058624}]
I20260620 00:17:55.919667 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.916787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45800 (local address 127.0.98.3:38445)
0620 00:17:55.916851 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.916853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.916901 (+ 48us) server_negotiation.cc:407] Connection header received
0620 00:17:55.916964 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.916965 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.916973 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.917023 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:55.917140 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.917681 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.918941 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.919149 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.919229 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.919416 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:55.919449 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:55.919488 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.919562 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.919580 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:17:55.925307 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.922526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45399 (local address 127.0.98.3:38445)
0620 00:17:55.922594 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:55.922595 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:55.922639 (+ 44us) server_negotiation.cc:407] Connection header received
0620 00:17:55.922689 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:55.922689 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:55.922701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:55.922769 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:55.922883 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.923557 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.924458 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.924734 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.924816 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.925019 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:55.925053 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:55.925113 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:55.925178 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 00:17:55.925200 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:17:55.925309 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:55.922392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38445 (local address 127.0.98.3:45399)
0620 00:17:55.922586 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:55.922596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:55.922646 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:55.922771 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:55.922771 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:55.922772 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:17:55.922824 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.922825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.923657 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.923659 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.924365 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:55.924367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:55.924851 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:55.924851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:55.924901 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:55.924916 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:17:55.925124 (+ 208us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:17:55.925126 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:17:55.925159 (+ 33us) client_negotiation.cc:240] Negotiation successful
0620 00:17:55.925195 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260620 00:17:55.928366 2276 raft_consensus.cc:1261] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Refusing update from remote peer 90d1c965abc3415fb35bc179549f0e6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:55.928366 2416 raft_consensus.cc:1261] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Refusing update from remote peer 90d1c965abc3415fb35bc179549f0e6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:55.928629 2664 consensus_queue.cc:1035] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:55.928783 2631 consensus_queue.cc:1035] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:55.942665 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:55.942792 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:55.943697 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318: Sending DeleteTablet for 3 replicas of tablet 7b9cabd75a1c407394f17dd6bd8c9c48
I20260620 00:17:55.943989 2529 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9cabd75a1c407394f17dd6bd8c9c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45768
I20260620 00:17:55.944041 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9cabd75a1c407394f17dd6bd8c9c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41298
I20260620 00:17:55.944332 2670 tablet_replica.cc:330] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: stopping tablet replica
I20260620 00:17:55.944469 2670 raft_consensus.cc:2229] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:55.944550 2670 raft_consensus.cc:2258] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:55.944550 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46598:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:17:55.944708 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:17:55.945039 2670 ts_tablet_manager.cc:1905] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:55.945288 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b9cabd75a1c407394f17dd6bd8c9c48 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:55 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48160
I20260620 00:17:55.945681 2671 tablet_replica.cc:330] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: stopping tablet replica
I20260620 00:17:55.946039 2256 tablet_service.cc:1467] Processing CreateTablet for tablet e9db5430c49d410fb02435d0adc6d5a7 (DEFAULT_TABLE table=test-table [id=e65c6bf1bfde43a4bc8b33be534b2b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:55.946164 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9db5430c49d410fb02435d0adc6d5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:55.946336 2396 tablet_service.cc:1467] Processing CreateTablet for tablet e9db5430c49d410fb02435d0adc6d5a7 (DEFAULT_TABLE table=test-table [id=e65c6bf1bfde43a4bc8b33be534b2b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:55.946446 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9db5430c49d410fb02435d0adc6d5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:55.947621 2670 ts_tablet_manager.cc:1918] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:17:55.947695 2670 log.cc:1196] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/7b9cabd75a1c407394f17dd6bd8c9c48
I20260620 00:17:55.947726 2529 tablet_service.cc:1467] Processing CreateTablet for tablet e9db5430c49d410fb02435d0adc6d5a7 (DEFAULT_TABLE table=test-table [id=e65c6bf1bfde43a4bc8b33be534b2b0f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:55.947811 2627 tablet_bootstrap.cc:492] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: Bootstrap starting.
I20260620 00:17:55.947835 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9db5430c49d410fb02435d0adc6d5a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:55.947995 2670 ts_tablet_manager.cc:1939] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting consensus metadata
I20260620 00:17:55.948242 2627 tablet_bootstrap.cc:654] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:55.948990 2105 catalog_manager.cc:4841] TS 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873): tablet 7b9cabd75a1c407394f17dd6bd8c9c48 (table test-table [id=5a603ea3c873493aaa34fc63f5323519]) successfully deleted
I20260620 00:17:55.949456 2626 tablet_bootstrap.cc:492] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: Bootstrap starting.
I20260620 00:17:55.949563 2625 tablet_bootstrap.cc:492] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: Bootstrap starting.
I20260620 00:17:55.949944 2626 tablet_bootstrap.cc:654] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:55.949966 2625 tablet_bootstrap.cc:654] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:55.950812 2671 raft_consensus.cc:2229] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:55.950924 2671 raft_consensus.cc:2258] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:55.951037 2625 tablet_bootstrap.cc:492] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: No bootstrap required, opened a new log
I20260620 00:17:55.951103 2625 ts_tablet_manager.cc:1397] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:17:55.951136 2669 tablet_replica.cc:330] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: stopping tablet replica
I20260620 00:17:55.951265 2669 raft_consensus.cc:2229] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:17:55.951256 2625 raft_consensus.cc:348] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.951329 2625 raft_consensus.cc:374] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:55.951359 2625 raft_consensus.cc:729] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c35ba0ff745a1bd3db537a4621e0c, State: Initialized, Role: FOLLOWER
I20260620 00:17:55.951408 2669 raft_consensus.cc:2258] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:55.951401 2625 consensus_queue.cc:260] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.951512 2625 ts_tablet_manager.cc:1428] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:55.952064 2626 tablet_bootstrap.cc:492] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: No bootstrap required, opened a new log
I20260620 00:17:55.952116 2626 ts_tablet_manager.cc:1397] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:17:55.952286 2626 raft_consensus.cc:348] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.952365 2626 raft_consensus.cc:374] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:55.952396 2626 raft_consensus.cc:729] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d1c965abc3415fb35bc179549f0e6f, State: Initialized, Role: FOLLOWER
I20260620 00:17:55.952438 2626 consensus_queue.cc:260] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.952543 2626 ts_tablet_manager.cc:1428] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:55.952504 2671 ts_tablet_manager.cc:1905] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:55.954280 2669 ts_tablet_manager.cc:1905] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:55.954408 2671 ts_tablet_manager.cc:1918] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:17:55.954501 2671 log.cc:1196] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/7b9cabd75a1c407394f17dd6bd8c9c48
I20260620 00:17:55.954790 2671 ts_tablet_manager.cc:1939] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 526c35ba0ff745a1bd3db537a4621e0c: Deleting consensus metadata
I20260620 00:17:55.955256 2106 catalog_manager.cc:4841] TS 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729): tablet 7b9cabd75a1c407394f17dd6bd8c9c48 (table test-table [id=5a603ea3c873493aaa34fc63f5323519]) successfully deleted
I20260620 00:17:55.955804 2669 ts_tablet_manager.cc:1918] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:17:55.955894 2669 log.cc:1196] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/7b9cabd75a1c407394f17dd6bd8c9c48
I20260620 00:17:55.956265 2669 ts_tablet_manager.cc:1939] T 7b9cabd75a1c407394f17dd6bd8c9c48 P 90d1c965abc3415fb35bc179549f0e6f: Deleting consensus metadata
I20260620 00:17:55.956557 2627 tablet_bootstrap.cc:492] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: No bootstrap required, opened a new log
I20260620 00:17:55.956622 2627 ts_tablet_manager.cc:1397] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260620 00:17:55.956751 2104 catalog_manager.cc:4841] TS 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3:38445): tablet 7b9cabd75a1c407394f17dd6bd8c9c48 (table test-table [id=5a603ea3c873493aaa34fc63f5323519]) successfully deleted
I20260620 00:17:55.956779 2627 raft_consensus.cc:348] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.956857 2627 raft_consensus.cc:374] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:55.956892 2627 raft_consensus.cc:729] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a9ca5e181b4eeebd1ee4ba5882589c, State: Initialized, Role: FOLLOWER
I20260620 00:17:55.956943 2627 consensus_queue.cc:260] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:55.957048 2627 ts_tablet_manager.cc:1428] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:56.332298 2632 raft_consensus.cc:482] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:17:56.332408 2632 raft_consensus.cc:504] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:56.332832 2632 leader_election.cc:288] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873), 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3:38445)
I20260620 00:17:56.336280 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:56.333120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40221 (local address 127.0.98.3:38445)
0620 00:17:56.333207 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:56.333209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:56.333292 (+ 83us) server_negotiation.cc:407] Connection header received
0620 00:17:56.333377 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:56.333378 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:56.333388 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:56.333439 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:56.333624 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.334179 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.335465 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.335876 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.335940 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:56.335950 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:56.336168 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 00:17:56.336212 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260620 00:17:56.336284 2675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:56.333081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38445 (local address 127.0.98.1:40221)
0620 00:17:56.333208 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:56.333220 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:56.333305 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:56.333484 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:56.333485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:56.333487 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:56.333554 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:56.333557 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.334270 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.334271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:56.335356 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:56.335359 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.335997 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.335998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:56.336066 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:56.336070 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:56.336123 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:17:56.336194 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":61,"threads_started":1}
I20260620 00:17:56.336536 2676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:56.333061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.98.1:58581)
0620 00:17:56.333202 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:56.333215 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:56.333297 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:56.333573 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:56.333574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:56.333575 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:56.333660 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:56.333662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.334268 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.334270 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:56.335380 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:56.335383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.336071 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.336072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:56.336313 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:56.336317 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:56.336379 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:17:56.336437 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":291,"thread_start_us":75,"threads_started":1}
I20260620 00:17:56.336596 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9db5430c49d410fb02435d0adc6d5a7" candidate_uuid: "526c35ba0ff745a1bd3db537a4621e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d1c965abc3415fb35bc179549f0e6f" is_pre_election: true
I20260620 00:17:56.336735 2677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:56.333160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58581 (local address 127.0.98.2:33873)
0620 00:17:56.333292 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:56.333295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:56.333305 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:17:56.333480 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:56.333481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:56.333493 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:56.333566 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:56.333729 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.334185 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.335540 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.335977 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.336058 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:56.336072 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:56.336530 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 00:17:56.336666 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
I20260620 00:17:56.336753 2556 raft_consensus.cc:2454] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526c35ba0ff745a1bd3db537a4621e0c in term 0.
I20260620 00:17:56.336922 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9db5430c49d410fb02435d0adc6d5a7" candidate_uuid: "526c35ba0ff745a1bd3db537a4621e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" is_pre_election: true
I20260620 00:17:56.337018 2416 raft_consensus.cc:2454] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 526c35ba0ff745a1bd3db537a4621e0c in term 0.
I20260620 00:17:56.336973 2208 leader_election.cc:302] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526c35ba0ff745a1bd3db537a4621e0c, 90d1c965abc3415fb35bc179549f0e6f; no voters:
I20260620 00:17:56.337122 2632 raft_consensus.cc:2781] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:17:56.337169 2632 raft_consensus.cc:482] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:17:56.337201 2632 raft_consensus.cc:3037] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:56.337918 2632 raft_consensus.cc:504] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:56.338068 2632 leader_election.cc:288] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [CANDIDATE]: Term 1 election: Requested vote from peers 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873), 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3:38445)
I20260620 00:17:56.338224 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9db5430c49d410fb02435d0adc6d5a7" candidate_uuid: "526c35ba0ff745a1bd3db537a4621e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d1c965abc3415fb35bc179549f0e6f"
I20260620 00:17:56.338234 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9db5430c49d410fb02435d0adc6d5a7" candidate_uuid: "526c35ba0ff745a1bd3db537a4621e0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c"
I20260620 00:17:56.338313 2416 raft_consensus.cc:3037] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:56.338316 2556 raft_consensus.cc:3037] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:56.338958 2416 raft_consensus.cc:2454] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526c35ba0ff745a1bd3db537a4621e0c in term 1.
I20260620 00:17:56.338968 2556 raft_consensus.cc:2454] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 526c35ba0ff745a1bd3db537a4621e0c in term 1.
I20260620 00:17:56.339123 2209 leader_election.cc:302] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a9ca5e181b4eeebd1ee4ba5882589c, 526c35ba0ff745a1bd3db537a4621e0c; no voters:
I20260620 00:17:56.339254 2632 raft_consensus.cc:2781] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:56.339395 2632 raft_consensus.cc:686] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 LEADER]: Becoming Leader. State: Replica: 526c35ba0ff745a1bd3db537a4621e0c, State: Running, Role: LEADER
I20260620 00:17:56.339505 2632 consensus_queue.cc:237] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:56.340269 2118 catalog_manager.cc:5495] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c reported cstate change: term changed from 0 to 1, leader changed from <none> to 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "526c35ba0ff745a1bd3db537a4621e0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } health_report { overall_health: UNKNOWN } } }
I20260620 00:17:56.395218 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:56.392599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48162 (local address 127.0.98.1:45729)
0620 00:17:56.392772 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:56.392774 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:56.392780 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:17:56.392813 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:56.392814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:56.392832 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:56.392913 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:56.393028 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.393560 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.394320 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.394602 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.394678 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:56.394825 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:56.394954 (+ 129us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:56.395037 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:56.395077 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 00:17:56.395097 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":100,"threads_started":1}
I20260620 00:17:56.400600 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:56.397879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40283 (local address 127.0.98.3:38445)
0620 00:17:56.397924 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:56.397926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:56.398056 (+ 130us) server_negotiation.cc:407] Connection header received
0620 00:17:56.398079 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:56.398080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:56.398091 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:56.398132 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:56.398252 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.398704 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.399470 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.399670 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.399980 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:56.400336 (+ 356us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:56.400366 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:56.400412 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:56.400505 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 00:17:56.400544 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:17:56.400615 2683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:56.397775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38445 (local address 127.0.98.1:40283)
0620 00:17:56.397923 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:56.397957 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:56.398011 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:56.398144 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:56.398145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:56.398146 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:17:56.398200 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:56.398201 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.398778 (+ 577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.398779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:56.399377 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:56.399379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.400025 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.400026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:56.400085 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:56.400098 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:17:56.400422 (+ 324us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:17:56.400423 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:17:56.400500 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:17:56.400539 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260620 00:17:56.403323 2556 raft_consensus.cc:1261] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Refusing update from remote peer 526c35ba0ff745a1bd3db537a4621e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:56.403537 2416 raft_consensus.cc:1261] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Refusing update from remote peer 526c35ba0ff745a1bd3db537a4621e0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:56.403633 2678 consensus_queue.cc:1035] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:56.403756 2632 consensus_queue.cc:1035] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:56.411091 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:56.408758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45729 (local address 127.0.98.3:42169)
0620 00:17:56.408807 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:56.408818 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:56.408851 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:56.409083 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:56.409085 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:56.409086 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:17:56.409130 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:56.409132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.409746 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.409746 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:56.410292 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:56.410294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.410703 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.410703 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:56.410758 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:56.410763 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:17:56.410935 (+ 172us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:17:56.410936 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:17:56.410985 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 00:17:56.411026 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 00:17:56.411088 2680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:56.408918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42169 (local address 127.0.98.1:45729)
0620 00:17:56.408972 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:56.408975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:56.408982 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:17:56.409005 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:56.409006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:56.409025 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:56.409078 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:56.409190 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.409667 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.410373 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:56.410637 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:56.410704 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:56.410849 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:56.410881 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:56.410933 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:56.410990 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 00:17:56.411009 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260620 00:17:56.411550 2653 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 00:17:57.935809 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:57.935911 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46598:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:57.936960 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318: Sending DeleteTablet for 3 replicas of tablet e9db5430c49d410fb02435d0adc6d5a7
I20260620 00:17:57.937250 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet e9db5430c49d410fb02435d0adc6d5a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41298
I20260620 00:17:57.937282 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet e9db5430c49d410fb02435d0adc6d5a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48160
I20260620 00:17:57.937343 2529 tablet_service.cc:1514] Processing DeleteTablet for tablet e9db5430c49d410fb02435d0adc6d5a7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45768
I20260620 00:17:57.937559 2709 tablet_replica.cc:330] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: stopping tablet replica
I20260620 00:17:57.937628 2709 raft_consensus.cc:2229] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:17:57.937747 2709 raft_consensus.cc:2258] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:57.937856 2711 tablet_replica.cc:330] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: stopping tablet replica
I20260620 00:17:57.937913 2711 raft_consensus.cc:2229] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:57.937949 2711 raft_consensus.cc:2258] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:57.938046 2709 ts_tablet_manager.cc:1905] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:57.938181 2711 ts_tablet_manager.cc:1905] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:57.939437 2709 ts_tablet_manager.cc:1918] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:17:57.939509 2709 log.cc:1196] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/e9db5430c49d410fb02435d0adc6d5a7
I20260620 00:17:57.939743 2709 ts_tablet_manager.cc:1939] T e9db5430c49d410fb02435d0adc6d5a7 P 526c35ba0ff745a1bd3db537a4621e0c: Deleting consensus metadata
I20260620 00:17:57.940068 2106 catalog_manager.cc:4841] TS 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729): tablet e9db5430c49d410fb02435d0adc6d5a7 (table test-table [id=e65c6bf1bfde43a4bc8b33be534b2b0f]) successfully deleted
I20260620 00:17:57.940191 2711 ts_tablet_manager.cc:1918] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:17:57.940261 2711 log.cc:1196] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/e9db5430c49d410fb02435d0adc6d5a7
I20260620 00:17:57.940357 2710 tablet_replica.cc:330] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: stopping tablet replica
I20260620 00:17:57.940428 2710 raft_consensus.cc:2229] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:57.940477 2710 raft_consensus.cc:2258] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:57.940510 2711 ts_tablet_manager.cc:1939] T e9db5430c49d410fb02435d0adc6d5a7 P 90d1c965abc3415fb35bc179549f0e6f: Deleting consensus metadata
I20260620 00:17:57.940735 2710 ts_tablet_manager.cc:1905] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:57.940951 2104 catalog_manager.cc:4841] TS 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3:38445): tablet e9db5430c49d410fb02435d0adc6d5a7 (table test-table [id=e65c6bf1bfde43a4bc8b33be534b2b0f]) successfully deleted
I20260620 00:17:57.942451 2710 ts_tablet_manager.cc:1918] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:17:57.942528 2710 log.cc:1196] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/e9db5430c49d410fb02435d0adc6d5a7
I20260620 00:17:57.942723 2710 ts_tablet_manager.cc:1939] T e9db5430c49d410fb02435d0adc6d5a7 P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting consensus metadata
I20260620 00:17:57.943255 2105 catalog_manager.cc:4841] TS 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873): tablet e9db5430c49d410fb02435d0adc6d5a7 (table test-table [id=e65c6bf1bfde43a4bc8b33be534b2b0f]) successfully deleted
Jun 20 00:17:57 dist-test-slave-r81b krb5kdc[2074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914677, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:17:57 dist-test-slave-r81b krb5kdc[2074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914677, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:17:57.969394 2719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:57.956089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46610 (local address 127.0.98.62:40035)
0620 00:17:57.956265 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:57.956267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:57.956276 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:17:57.956331 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:57.956331 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:57.956343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:57.956404 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:17:57.957002 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:57.957553 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:57.958340 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:57.958561 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:57.958615 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:57.966391 (+ 7776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:17:57.966393 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:17:57.966393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:17:57.966402 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:17:57.968041 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:57.968457 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:57.968458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:57.968459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:57.968561 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:17:57.968800 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:17:57.968801 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:17:57.968801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:17:57.968924 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:17:57.969037 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:57.969183 (+ 146us) server_negotiation.cc:299] Negotiation successful
0620 00:17:57.969274 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":126,"threads_started":1}
I20260620 00:17:57.971254 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:17:57.971457 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:17:57.972908 2396 tablet_service.cc:1467] Processing CreateTablet for tablet bd9867b7d48749f98c59e223a0cebb7d (DEFAULT_TABLE table=test-table [id=572341dbab7b43ad9445be5434f3229f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:57.972895 2256 tablet_service.cc:1467] Processing CreateTablet for tablet bd9867b7d48749f98c59e223a0cebb7d (DEFAULT_TABLE table=test-table [id=572341dbab7b43ad9445be5434f3229f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:57.972998 2529 tablet_service.cc:1467] Processing CreateTablet for tablet bd9867b7d48749f98c59e223a0cebb7d (DEFAULT_TABLE table=test-table [id=572341dbab7b43ad9445be5434f3229f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:57.973060 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9867b7d48749f98c59e223a0cebb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:57.973066 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9867b7d48749f98c59e223a0cebb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:57.973142 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd9867b7d48749f98c59e223a0cebb7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:57.974275 2720 tablet_bootstrap.cc:492] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: Bootstrap starting.
I20260620 00:17:57.974488 2721 tablet_bootstrap.cc:492] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: Bootstrap starting.
I20260620 00:17:57.974646 2722 tablet_bootstrap.cc:492] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: Bootstrap starting.
I20260620 00:17:57.974853 2720 tablet_bootstrap.cc:654] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:57.974879 2721 tablet_bootstrap.cc:654] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:57.974956 2722 tablet_bootstrap.cc:654] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:57.975481 2720 tablet_bootstrap.cc:492] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: No bootstrap required, opened a new log
I20260620 00:17:57.975495 2721 tablet_bootstrap.cc:492] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: No bootstrap required, opened a new log
I20260620 00:17:57.975544 2720 ts_tablet_manager.cc:1397] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:57.975540 2721 ts_tablet_manager.cc:1397] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:57.975663 2720 raft_consensus.cc:348] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:57.975672 2721 raft_consensus.cc:348] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:57.975739 2720 raft_consensus.cc:374] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:57.975768 2720 raft_consensus.cc:729] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d1c965abc3415fb35bc179549f0e6f, State: Initialized, Role: FOLLOWER
I20260620 00:17:57.975769 2721 raft_consensus.cc:374] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:57.975813 2721 raft_consensus.cc:729] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a9ca5e181b4eeebd1ee4ba5882589c, State: Initialized, Role: FOLLOWER
I20260620 00:17:57.975818 2720 consensus_queue.cc:260] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:57.975854 2721 consensus_queue.cc:260] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:57.975925 2720 ts_tablet_manager.cc:1428] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:17:57.976014 2721 ts_tablet_manager.cc:1428] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:17:57.976248 2722 tablet_bootstrap.cc:492] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: No bootstrap required, opened a new log
I20260620 00:17:57.976297 2722 ts_tablet_manager.cc:1397] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:17:57.976408 2722 raft_consensus.cc:348] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:57.976593 2722 raft_consensus.cc:374] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:57.976621 2722 raft_consensus.cc:729] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c35ba0ff745a1bd3db537a4621e0c, State: Initialized, Role: FOLLOWER
I20260620 00:17:57.976684 2722 consensus_queue.cc:260] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:57.976853 2722 ts_tablet_manager.cc:1428] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:58.102541 2695 raft_consensus.cc:482] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:17:58.102655 2695 raft_consensus.cc:504] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.102859 2695 leader_election.cc:288] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729), 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873)
I20260620 00:17:58.103123 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9867b7d48749f98c59e223a0cebb7d" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" is_pre_election: true
I20260620 00:17:58.103129 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9867b7d48749f98c59e223a0cebb7d" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c35ba0ff745a1bd3db537a4621e0c" is_pre_election: true
I20260620 00:17:58.103232 2416 raft_consensus.cc:2454] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 0.
I20260620 00:17:58.103235 2276 raft_consensus.cc:2454] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 0.
I20260620 00:17:58.103420 2490 leader_election.cc:302] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 526c35ba0ff745a1bd3db537a4621e0c, 90d1c965abc3415fb35bc179549f0e6f; no voters:
I20260620 00:17:58.103564 2695 raft_consensus.cc:2781] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:17:58.103600 2695 raft_consensus.cc:482] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:17:58.103627 2695 raft_consensus.cc:3037] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:58.104439 2695 raft_consensus.cc:504] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.104584 2695 leader_election.cc:288] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 election: Requested vote from peers 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729), 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873)
I20260620 00:17:58.104743 2416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9867b7d48749f98c59e223a0cebb7d" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c"
I20260620 00:17:58.104739 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd9867b7d48749f98c59e223a0cebb7d" candidate_uuid: "90d1c965abc3415fb35bc179549f0e6f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c35ba0ff745a1bd3db537a4621e0c"
I20260620 00:17:58.104818 2416 raft_consensus.cc:3037] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:58.104837 2276 raft_consensus.cc:3037] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:58.105444 2416 raft_consensus.cc:2454] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 1.
I20260620 00:17:58.105631 2489 leader_election.cc:302] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a9ca5e181b4eeebd1ee4ba5882589c, 90d1c965abc3415fb35bc179549f0e6f; no voters:
I20260620 00:17:58.105666 2276 raft_consensus.cc:2454] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90d1c965abc3415fb35bc179549f0e6f in term 1.
I20260620 00:17:58.105743 2695 raft_consensus.cc:2781] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:58.105808 2695 raft_consensus.cc:686] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 1 LEADER]: Becoming Leader. State: Replica: 90d1c965abc3415fb35bc179549f0e6f, State: Running, Role: LEADER
I20260620 00:17:58.105861 2695 consensus_queue.cc:237] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.106536 2118 catalog_manager.cc:5495] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f reported cstate change: term changed from 0 to 1, leader changed from <none> to 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3). New cstate: current_term: 1 leader_uuid: "90d1c965abc3415fb35bc179549f0e6f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } health_report { overall_health: HEALTHY } } }
I20260620 00:17:58.126394 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.123950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45812 (local address 127.0.98.3:38445)
0620 00:17:58.124119 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:58.124121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:58.124129 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:17:58.124184 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:58.124185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:58.124196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:58.124271 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:58.124384 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.124926 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.125702 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.125941 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.126006 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.126128 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:58.126158 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:58.126201 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:58.126252 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 00:17:58.126271 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":109,"threads_started":1}
I20260620 00:17:58.128886 2276 raft_consensus.cc:1261] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Refusing update from remote peer 90d1c965abc3415fb35bc179549f0e6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:58.128881 2416 raft_consensus.cc:1261] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Refusing update from remote peer 90d1c965abc3415fb35bc179549f0e6f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:58.129142 2690 consensus_queue.cc:1035] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:58.129272 2664 consensus_queue.cc:1035] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [LEADER]: Connected to new peer: Peer: permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:58.140355 2119 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:58.140530 2119 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:58.141234 2119 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318: Sending DeleteTablet for 3 replicas of tablet bd9867b7d48749f98c59e223a0cebb7d
I20260620 00:17:58.141502 2529 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9867b7d48749f98c59e223a0cebb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45768
I20260620 00:17:58.141512 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9867b7d48749f98c59e223a0cebb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48160
I20260620 00:17:58.141538 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet bd9867b7d48749f98c59e223a0cebb7d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:58 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41298
I20260620 00:17:58.141608 2711 tablet_replica.cc:330] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: stopping tablet replica
I20260620 00:17:58.141618 2710 tablet_replica.cc:330] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: stopping tablet replica
I20260620 00:17:58.141664 2710 raft_consensus.cc:2229] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:58.141661 2711 raft_consensus.cc:2229] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:17:58.141697 2710 raft_consensus.cc:2258] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:58.141721 2711 raft_consensus.cc:2258] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:58.141726 2709 tablet_replica.cc:330] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: stopping tablet replica
I20260620 00:17:58.141784 2709 raft_consensus.cc:2229] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:58.141824 2709 raft_consensus.cc:2258] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:58.142027 2710 ts_tablet_manager.cc:1905] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:58.142050 2711 ts_tablet_manager.cc:1905] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:58.142153 2709 ts_tablet_manager.cc:1905] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:58.143548 2710 ts_tablet_manager.cc:1918] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:17:58.143654 2710 log.cc:1196] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/bd9867b7d48749f98c59e223a0cebb7d
I20260620 00:17:58.143647 2709 ts_tablet_manager.cc:1918] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:17:58.143647 2711 ts_tablet_manager.cc:1918] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:17:58.143704 2711 log.cc:1196] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/bd9867b7d48749f98c59e223a0cebb7d
I20260620 00:17:58.143704 2709 log.cc:1196] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/bd9867b7d48749f98c59e223a0cebb7d
I20260620 00:17:58.143890 2710 ts_tablet_manager.cc:1939] T bd9867b7d48749f98c59e223a0cebb7d P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting consensus metadata
I20260620 00:17:58.143940 2709 ts_tablet_manager.cc:1939] T bd9867b7d48749f98c59e223a0cebb7d P 526c35ba0ff745a1bd3db537a4621e0c: Deleting consensus metadata
I20260620 00:17:58.143940 2711 ts_tablet_manager.cc:1939] T bd9867b7d48749f98c59e223a0cebb7d P 90d1c965abc3415fb35bc179549f0e6f: Deleting consensus metadata
I20260620 00:17:58.144399 2105 catalog_manager.cc:4841] TS 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873): tablet bd9867b7d48749f98c59e223a0cebb7d (table test-table [id=572341dbab7b43ad9445be5434f3229f]) successfully deleted
I20260620 00:17:58.144440 2104 catalog_manager.cc:4841] TS 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3:38445): tablet bd9867b7d48749f98c59e223a0cebb7d (table test-table [id=572341dbab7b43ad9445be5434f3229f]) successfully deleted
I20260620 00:17:58.144493 2106 catalog_manager.cc:4841] TS 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729): tablet bd9867b7d48749f98c59e223a0cebb7d (table test-table [id=572341dbab7b43ad9445be5434f3229f]) successfully deleted
I20260620 00:17:58.144622 2119 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:17:58.144769 2119 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:17:58.146086 2396 tablet_service.cc:1467] Processing CreateTablet for tablet 36245bd1c1a34b808c2e89f636bc0034 (DEFAULT_TABLE table=test-table [id=2eeb5c4025e24b5b9c65f77e7568a7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:58.146092 2256 tablet_service.cc:1467] Processing CreateTablet for tablet 36245bd1c1a34b808c2e89f636bc0034 (DEFAULT_TABLE table=test-table [id=2eeb5c4025e24b5b9c65f77e7568a7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:58.146095 2529 tablet_service.cc:1467] Processing CreateTablet for tablet 36245bd1c1a34b808c2e89f636bc0034 (DEFAULT_TABLE table=test-table [id=2eeb5c4025e24b5b9c65f77e7568a7b8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:17:58.146195 2396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36245bd1c1a34b808c2e89f636bc0034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:58.146203 2529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36245bd1c1a34b808c2e89f636bc0034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:58.146203 2256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36245bd1c1a34b808c2e89f636bc0034. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:17:58.147049 2721 tablet_bootstrap.cc:492] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: Bootstrap starting.
I20260620 00:17:58.147413 2722 tablet_bootstrap.cc:492] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c: Bootstrap starting.
I20260620 00:17:58.147774 2722 tablet_bootstrap.cc:654] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:58.147774 2721 tablet_bootstrap.cc:654] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:58.148161 2720 tablet_bootstrap.cc:492] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: Bootstrap starting.
I20260620 00:17:58.148406 2722 tablet_bootstrap.cc:492] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c: No bootstrap required, opened a new log
I20260620 00:17:58.148453 2722 ts_tablet_manager.cc:1397] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:17:58.148506 2720 tablet_bootstrap.cc:654] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: Neither blocks nor log segments found. Creating new log.
I20260620 00:17:58.148713 2722 raft_consensus.cc:348] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.148806 2722 raft_consensus.cc:374] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:58.148838 2722 raft_consensus.cc:729] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 526c35ba0ff745a1bd3db537a4621e0c, State: Initialized, Role: FOLLOWER
I20260620 00:17:58.148883 2722 consensus_queue.cc:260] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.148984 2722 ts_tablet_manager.cc:1428] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 00:17:58.149065 2720 tablet_bootstrap.cc:492] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: No bootstrap required, opened a new log
I20260620 00:17:58.149114 2720 ts_tablet_manager.cc:1397] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:17:58.149236 2720 raft_consensus.cc:348] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.149291 2720 raft_consensus.cc:374] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:58.149304 2720 raft_consensus.cc:729] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90d1c965abc3415fb35bc179549f0e6f, State: Initialized, Role: FOLLOWER
I20260620 00:17:58.149353 2720 consensus_queue.cc:260] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.149446 2720 ts_tablet_manager.cc:1428] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:58.149730 2721 tablet_bootstrap.cc:492] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: No bootstrap required, opened a new log
I20260620 00:17:58.149807 2721 ts_tablet_manager.cc:1397] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:17:58.149969 2721 raft_consensus.cc:348] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.150067 2721 raft_consensus.cc:374] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:17:58.150099 2721 raft_consensus.cc:729] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12a9ca5e181b4eeebd1ee4ba5882589c, State: Initialized, Role: FOLLOWER
I20260620 00:17:58.150151 2721 consensus_queue.cc:260] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.150254 2721 ts_tablet_manager.cc:1428] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:17:58.317063 2726 raft_consensus.cc:482] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:17:58.317173 2726 raft_consensus.cc:504] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.317582 2726 leader_election.cc:288] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729), 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3:38445)
I20260620 00:17:58.320648 2739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.317868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57605 (local address 127.0.98.1:45729)
0620 00:17:58.317978 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:58.318004 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:58.318018 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:17:58.318102 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:58.318103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:58.318117 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:58.318202 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:58.318352 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.318847 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.319822 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.320283 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.320370 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.320383 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:58.320534 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 00:17:58.320579 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
I20260620 00:17:58.320665 2737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.317796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45729 (local address 127.0.98.2:57605)
0620 00:17:58.317956 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:58.317969 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:58.318039 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:58.318203 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:58.318203 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:58.318206 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:58.318282 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:58.318283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.318941 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.318942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:58.319732 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:58.319733 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.320372 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.320373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:58.320445 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.320449 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:58.320523 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:17:58.320568 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":1,"thread_start_us":76,"threads_started":1}
I20260620 00:17:58.320767 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.317974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50569 (local address 127.0.98.3:38445)
0620 00:17:58.318019 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:58.318021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:58.318031 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:17:58.318121 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:58.318122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:58.318132 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:58.318198 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:17:58.318357 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.319023 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.320002 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.320370 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.320475 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.320486 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:58.320664 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 00:17:58.320705 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:17:58.320782 2738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.317801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38445 (local address 127.0.98.2:50569)
0620 00:17:58.317978 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:58.317989 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:58.318046 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:58.318201 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:58.318202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:58.318204 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:17:58.318288 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:58.318290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.319152 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.319154 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:58.319900 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:58.319902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.320524 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.320525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:58.320588 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.320592 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:17:58.320657 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:17:58.320698 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":3,"thread_start_us":75,"threads_started":1}
I20260620 00:17:58.321106 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36245bd1c1a34b808c2e89f636bc0034" candidate_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d1c965abc3415fb35bc179549f0e6f" is_pre_election: true
I20260620 00:17:58.321106 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36245bd1c1a34b808c2e89f636bc0034" candidate_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c35ba0ff745a1bd3db537a4621e0c" is_pre_election: true
I20260620 00:17:58.321204 2276 raft_consensus.cc:2454] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a9ca5e181b4eeebd1ee4ba5882589c in term 0.
I20260620 00:17:58.321204 2556 raft_consensus.cc:2454] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12a9ca5e181b4eeebd1ee4ba5882589c in term 0.
I20260620 00:17:58.321406 2348 leader_election.cc:302] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12a9ca5e181b4eeebd1ee4ba5882589c, 526c35ba0ff745a1bd3db537a4621e0c, 90d1c965abc3415fb35bc179549f0e6f; no voters:
I20260620 00:17:58.321558 2726 raft_consensus.cc:2781] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:17:58.321610 2726 raft_consensus.cc:482] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:17:58.321637 2726 raft_consensus.cc:3037] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:58.322327 2726 raft_consensus.cc:504] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.322474 2726 leader_election.cc:288] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [CANDIDATE]: Term 1 election: Requested vote from peers 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729), 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3:38445)
I20260620 00:17:58.322608 2276 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36245bd1c1a34b808c2e89f636bc0034" candidate_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "526c35ba0ff745a1bd3db537a4621e0c"
I20260620 00:17:58.322629 2556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36245bd1c1a34b808c2e89f636bc0034" candidate_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90d1c965abc3415fb35bc179549f0e6f"
I20260620 00:17:58.322675 2276 raft_consensus.cc:3037] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:58.322701 2556 raft_consensus.cc:3037] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:17:58.323267 2276 raft_consensus.cc:2454] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a9ca5e181b4eeebd1ee4ba5882589c in term 1.
I20260620 00:17:58.323284 2556 raft_consensus.cc:2454] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12a9ca5e181b4eeebd1ee4ba5882589c in term 1.
I20260620 00:17:58.323444 2350 leader_election.cc:302] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12a9ca5e181b4eeebd1ee4ba5882589c, 526c35ba0ff745a1bd3db537a4621e0c; no voters:
I20260620 00:17:58.323565 2726 raft_consensus.cc:2781] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:17:58.323742 2726 raft_consensus.cc:686] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 LEADER]: Becoming Leader. State: Replica: 12a9ca5e181b4eeebd1ee4ba5882589c, State: Running, Role: LEADER
I20260620 00:17:58.323822 2726 consensus_queue.cc:237] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } }
I20260620 00:17:58.324508 2118 catalog_manager.cc:5495] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c reported cstate change: term changed from 0 to 1, leader changed from <none> to 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12a9ca5e181b4eeebd1ee4ba5882589c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33873 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 } health_report { overall_health: UNKNOWN } } }
I20260620 00:17:58.333698 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.331136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41314 (local address 127.0.98.2:33873)
0620 00:17:58.331299 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:58.331301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:58.331308 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:17:58.331356 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:58.331356 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:58.331374 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:58.331473 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:58.331578 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.332070 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.332838 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.333091 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.333169 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.333324 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:58.333451 (+ 127us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:58.333534 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:58.333572 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 00:17:58.333592 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":91,"threads_started":1}
I20260620 00:17:58.338882 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.336321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47767 (local address 127.0.98.3:38445)
0620 00:17:58.336370 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:58.336372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:58.336545 (+ 173us) server_negotiation.cc:407] Connection header received
0620 00:17:58.336588 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:58.336589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:58.336601 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:58.336646 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:58.336777 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.337277 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.338093 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.338293 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.338402 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.338626 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:58.338658 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:58.338709 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:58.338800 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 00:17:58.338822 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:17:58.338905 2745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.336252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38445 (local address 127.0.98.2:47767)
0620 00:17:58.336476 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:58.336486 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:58.336544 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:58.336647 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:58.336648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:58.336649 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:17:58.336701 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:58.336702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.337357 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.337358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:58.337993 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:58.337995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.338455 (+ 460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.338456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:58.338508 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.338522 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:17:58.338738 (+ 216us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:17:58.338741 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:17:58.338788 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:17:58.338830 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":82,"threads_started":1}
I20260620 00:17:58.341416 2276 raft_consensus.cc:1261] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Refusing update from remote peer 12a9ca5e181b4eeebd1ee4ba5882589c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:58.341456 2556 raft_consensus.cc:1261] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Refusing update from remote peer 12a9ca5e181b4eeebd1ee4ba5882589c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:17:58.341704 2740 consensus_queue.cc:1035] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [LEADER]: Connected to new peer: Peer: permanent_uuid: "90d1c965abc3415fb35bc179549f0e6f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38445 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:58.341838 2726 consensus_queue.cc:1035] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [LEADER]: Connected to new peer: Peer: permanent_uuid: "526c35ba0ff745a1bd3db537a4621e0c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45729 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:17:58.348820 2742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.346339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59731 (local address 127.0.98.2:33873)
0620 00:17:58.346376 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:17:58.346378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:17:58.346462 (+ 84us) server_negotiation.cc:407] Connection header received
0620 00:17:58.346597 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:17:58.346598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:17:58.346609 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:17:58.346660 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:17:58.346770 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.347215 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.348086 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.348407 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.348507 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.348642 (+ 135us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:17:58.348669 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:17:58.348702 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 00:17:58.348733 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 00:17:58.348750 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 00:17:58.348866 2749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:58.346263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33873 (local address 127.0.98.3:59731)
0620 00:17:58.346401 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:58.346411 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:17:58.346490 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:17:58.346675 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:17:58.346675 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:17:58.346678 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:17:58.346728 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:58.346730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.347270 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.347270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:58.348003 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:17:58.348006 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:17:58.348545 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:17:58.348546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:17:58.348589 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:17:58.348593 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:17:58.348701 (+ 108us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:17:58.348702 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:17:58.348731 (+ 29us) client_negotiation.cc:240] Negotiation successful
0620 00:17:58.348779 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":95,"threads_started":1}
W20260620 00:17:58.349162 2691 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260620 00:17:59.872411 2118 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:59.872526 2118 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:17:59.873430 2118 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b92b9fd95f2e4405b9d668f29ea97318: Sending DeleteTablet for 3 replicas of tablet 36245bd1c1a34b808c2e89f636bc0034
I20260620 00:17:59.873693 2256 tablet_service.cc:1514] Processing DeleteTablet for tablet 36245bd1c1a34b808c2e89f636bc0034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48160
I20260620 00:17:59.873783 2529 tablet_service.cc:1514] Processing DeleteTablet for tablet 36245bd1c1a34b808c2e89f636bc0034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45768
I20260620 00:17:59.873940 2773 tablet_replica.cc:330] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c: stopping tablet replica
I20260620 00:17:59.874027 2773 raft_consensus.cc:2229] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:59.874045 2774 tablet_replica.cc:330] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: stopping tablet replica
I20260620 00:17:59.874094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 2191
I20260620 00:17:59.874109 2773 raft_consensus.cc:2258] T 36245bd1c1a34b808c2e89f636bc0034 P 526c35ba0ff745a1bd3db537a4621e0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:59.874116 2774 raft_consensus.cc:2229] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:17:59.874553 2396 tablet_service.cc:1514] Processing DeleteTablet for tablet 36245bd1c1a34b808c2e89f636bc0034 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:17:59 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41298
I20260620 00:17:59.874742 2775 tablet_replica.cc:330] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: stopping tablet replica
I20260620 00:17:59.874816 2775 raft_consensus.cc:2229] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:17:59.874929 2775 raft_consensus.cc:2258] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:59.875188 2775 ts_tablet_manager.cc:1905] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:59.874166 2774 raft_consensus.cc:2258] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:17:59.875470 2774 ts_tablet_manager.cc:1905] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:17:59.877051 2775 ts_tablet_manager.cc:1918] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:17:59.877111 2775 log.cc:1196] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/36245bd1c1a34b808c2e89f636bc0034
I20260620 00:17:59.877349 2775 ts_tablet_manager.cc:1939] T 36245bd1c1a34b808c2e89f636bc0034 P 12a9ca5e181b4eeebd1ee4ba5882589c: Deleting consensus metadata
I20260620 00:17:59.877439 2774 ts_tablet_manager.cc:1918] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260620 00:17:59.877509 2774 log.cc:1196] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/36245bd1c1a34b808c2e89f636bc0034
I20260620 00:17:59.877789 2774 ts_tablet_manager.cc:1939] T 36245bd1c1a34b808c2e89f636bc0034 P 90d1c965abc3415fb35bc179549f0e6f: Deleting consensus metadata
I20260620 00:17:59.877784 2105 catalog_manager.cc:4841] TS 12a9ca5e181b4eeebd1ee4ba5882589c (127.0.98.2:33873): tablet 36245bd1c1a34b808c2e89f636bc0034 (table test-table [id=2eeb5c4025e24b5b9c65f77e7568a7b8]) successfully deleted
I20260620 00:17:59.878123 2104 catalog_manager.cc:4841] TS 90d1c965abc3415fb35bc179549f0e6f (127.0.98.3:38445): tablet 36245bd1c1a34b808c2e89f636bc0034 (table test-table [id=2eeb5c4025e24b5b9c65f77e7568a7b8]) successfully deleted
W20260620 00:17:59.881307 2106 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45729 (error 108)
I20260620 00:17:59.881455 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 2331
I20260620 00:17:59.883971 2777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:17:59.883735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45729 (local address 127.0.0.1:48166)
0620 00:17:59.883871 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0620 00:17:59.883925 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45729: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
W20260620 00:17:59.884124 2106 catalog_manager.cc:4584] TS 526c35ba0ff745a1bd3db537a4621e0c (127.0.98.1:45729): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 36245bd1c1a34b808c2e89f636bc0034: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45729: connect: Connection refused (error 111)
I20260620 00:17:59.887147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 2471
I20260620 00:17:59.893072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 2087
2026-06-20T00:17:59Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6525 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:17:59 dist-test-slave-r81b krb5kdc[2780](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:17:59 dist-test-slave-r81b krb5kdc[2780](info): set up 2 sockets
Jun 20 00:17:59 dist-test-slave-r81b krb5kdc[2780](info): commencing operation
krb5kdc: starting...
W20260620 00:18:00.976086 392 mini_kdc.cc:121] Time spent starting KDC: real 1.061s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:18:01.095531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44567
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44567
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:01.179308 2793 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:01.179540 2793 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:01.179565 2793 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:01.180968 2793 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:01.181021 2793 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:01.181036 2793 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:01.181051 2793 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:01.181062 2793 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:01.182761 2793 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44567
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44567
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2793
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:01.183074 2793 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:01.183372 2793 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:01.188822 2798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:01.188874 2799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:01.188884 2801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:01.189185 2793 server_base.cc:1034] running on GCE node
I20260620 00:18:01.189510 2793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:01.189842 2793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:01.191008 2793 hybrid_clock.cc:648] HybridClock initialized: now 1781914681190989 us; error 35 us; skew 500 ppm
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:01.193869 2793 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:18:01.194413 2793 webserver.cc:460] Webserver started at http://127.0.98.62:34647/ using document root <none> and password file <none>
I20260620 00:18:01.194664 2793 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:01.194737 2793 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:01.194885 2793 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:01.196053 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "ff92f8cfb1cd40d59083bf03ac3741ef"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "20b5d1c229eda9eb40f8c011bda83418"
server_key_iv: "58e53897c987c8892db0dbc1233cd59c"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.196552 2793 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "ff92f8cfb1cd40d59083bf03ac3741ef"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "20b5d1c229eda9eb40f8c011bda83418"
server_key_iv: "58e53897c987c8892db0dbc1233cd59c"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.198779 2793 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 00:18:01.200098 2808 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.200340 2793 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:01.200464 2793 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "ff92f8cfb1cd40d59083bf03ac3741ef"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "20b5d1c229eda9eb40f8c011bda83418"
server_key_iv: "58e53897c987c8892db0dbc1233cd59c"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.200560 2793 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:01.216226 2793 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:01.218724 2793 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:01.218890 2793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:01.223009 2793 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44567
I20260620 00:18:01.223042 2860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44567 every 8 connection(s)
I20260620 00:18:01.223564 2793 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:01.224385 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:01.226655 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef: Bootstrap starting.
I20260620 00:18:01.227238 2861 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:01.227524 2861 log.cc:824] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:01.228349 2861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef: No bootstrap required, opened a new log
I20260620 00:18:01.229967 2861 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff92f8cfb1cd40d59083bf03ac3741ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44567 } }
I20260620 00:18:01.230114 2861 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:01.230139 2861 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff92f8cfb1cd40d59083bf03ac3741ef, State: Initialized, Role: FOLLOWER
I20260620 00:18:01.230320 2861 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff92f8cfb1cd40d59083bf03ac3741ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44567 } }
I20260620 00:18:01.230384 2861 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:01.230406 2861 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:01.230453 2861 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:01.230686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 2793
I20260620 00:18:01.230760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:01.230945 392 external_mini_cluster.cc:1428] Setting key 0a9ffbe803c783c16ad2ea3b97821e32
I20260620 00:18:01.231566 2861 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff92f8cfb1cd40d59083bf03ac3741ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44567 } }
I20260620 00:18:01.231688 2861 leader_election.cc:302] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff92f8cfb1cd40d59083bf03ac3741ef; no voters:
I20260620 00:18:01.231869 2861 leader_election.cc:288] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:01.231901 2866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:01.232007 2866 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [term 1 LEADER]: Becoming Leader. State: Replica: ff92f8cfb1cd40d59083bf03ac3741ef, State: Running, Role: LEADER
I20260620 00:18:01.232154 2861 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:01.232175 2866 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff92f8cfb1cd40d59083bf03ac3741ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44567 } }
I20260620 00:18:01.232576 2868 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff92f8cfb1cd40d59083bf03ac3741ef. Latest consensus state: current_term: 1 leader_uuid: "ff92f8cfb1cd40d59083bf03ac3741ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff92f8cfb1cd40d59083bf03ac3741ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44567 } } }
I20260620 00:18:01.232708 2868 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:01.232581 2867 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff92f8cfb1cd40d59083bf03ac3741ef" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff92f8cfb1cd40d59083bf03ac3741ef" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44567 } } }
I20260620 00:18:01.232805 2867 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:01.233140 2874 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:01.233704 2874 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:01.236181 2874 catalog_manager.cc:1347] Generated new cluster ID: e7e448f099da4d069240a7f726e584d8
I20260620 00:18:01.236270 2874 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:01.241976 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:01.231464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.0.98.62:44567)
0620 00:18:01.231641 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:01.231651 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:01.231686 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:18:01.231822 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:01.231830 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:01.231880 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:01.232060 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:01.233037 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.234592 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.235661 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.236304 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.236416 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:01.238804 (+ 2388us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:01.238808 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:01.238808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:01.238819 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:01.240508 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:01.240873 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:01.240876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:01.240877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:01.240970 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:01.241166 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:01.241167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:01.241167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:01.241408 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:01.241599 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:01.241664 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 00:18:01.241815 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":100,"threads_started":1}
I20260620 00:18:01.258908 2874 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:01.259665 2874 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:01.264075 2874 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff92f8cfb1cd40d59083bf03ac3741ef: Generated new TSK 0
I20260620 00:18:01.264336 2874 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:18:01.295037 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44567
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:01.412933 2887 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:01.413163 2887 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:01.413197 2887 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:01.414543 2887 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:01.414583 2887 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:01.414614 2887 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:01.416368 2887 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2887
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:01.416625 2887 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:01.416869 2887 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:01.422432 2892 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:01.422461 2893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:01.422461 2895 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:01.422705 2887 server_base.cc:1034] running on GCE node
I20260620 00:18:01.422920 2887 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:01.423161 2887 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:01.424309 2887 hybrid_clock.cc:648] HybridClock initialized: now 1781914681424291 us; error 35 us; skew 500 ppm
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:01.426779 2887 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:18:01.427246 2887 webserver.cc:460] Webserver started at http://127.0.98.1:37593/ using document root <none> and password file <none>
I20260620 00:18:01.427464 2887 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:01.427527 2887 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:01.427668 2887 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:01.428828 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "437e3f6f1edf49a8b65cc209fdb50e9a"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "4aece448a297a64e7e40ea09b8415fba"
server_key_iv: "7e06bea315e5c5dc0d8fa475e0d5acd5"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.429181 2887 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "437e3f6f1edf49a8b65cc209fdb50e9a"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "4aece448a297a64e7e40ea09b8415fba"
server_key_iv: "7e06bea315e5c5dc0d8fa475e0d5acd5"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.430477 2887 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:01.431222 2902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.431416 2887 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:01.431510 2887 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "437e3f6f1edf49a8b65cc209fdb50e9a"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "4aece448a297a64e7e40ea09b8415fba"
server_key_iv: "7e06bea315e5c5dc0d8fa475e0d5acd5"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.431586 2887 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:01.440487 2887 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:01.442404 2887 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:01.442557 2887 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:01.442785 2887 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:01.443082 2887 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:01.443135 2887 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.443176 2887 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:01.443210 2887 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.448957 2887 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42575
I20260620 00:18:01.449000 3015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42575 every 8 connection(s)
I20260620 00:18:01.449308 2887 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:18:01.450995 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 2887
I20260620 00:18:01.451061 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:01.451320 392 external_mini_cluster.cc:1428] Setting key 60c6ce6288bd8c64546ac023926b7590
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:18:01.459297 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:01.449985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38681 (local address 127.0.98.62:44567)
0620 00:18:01.450048 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:01.450051 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:01.450251 (+ 200us) server_negotiation.cc:407] Connection header received
0620 00:18:01.451232 (+ 981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:01.451234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:01.451247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:01.451315 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:01.452113 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.452918 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.453992 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.454202 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.454290 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:01.456478 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:01.456479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:01.456480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:01.456494 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:01.457885 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:01.458307 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:01.458309 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:01.458309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:01.458367 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:01.458611 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:01.458613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:01.458613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:01.458731 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:01.458832 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:01.459114 (+ 282us) server_negotiation.cc:299] Negotiation successful
0620 00:18:01.459202 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:18:01.459515 3018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:01.449970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44567 (local address 127.0.98.1:38681)
0620 00:18:01.450147 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:01.450183 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:01.451139 (+ 956us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:01.451412 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:01.451417 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:01.451759 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:01.452021 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:01.452027 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.453002 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.453003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:01.453897 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:01.453900 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.454303 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.454304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:01.454431 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:01.454957 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:01.454965 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:01.456381 (+ 1416us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:01.457983 (+ 1602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:01.457990 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:01.457993 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:01.458221 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:01.458431 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:01.458432 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:01.458432 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:01.458547 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:01.458814 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:01.458821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:01.458985 (+ 164us) client_negotiation.cc:769] Sending connection context
0620 00:18:01.459092 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 00:18:01.459257 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
I20260620 00:18:01.460029 3016 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44567
I20260620 00:18:01.460132 3016 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:01.460332 3016 heartbeater.cc:511] Master 127.0.98.62:44567 requested a full tablet report, sending...
I20260620 00:18:01.461054 2825 ts_manager.cc:194] Registered new tserver with Master: 437e3f6f1edf49a8b65cc209fdb50e9a (127.0.98.1:42575)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:18:01.461735 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:18:01.477274 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44567
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:01.559237 3021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:01.559561 3021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:01.559598 3021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:01.560904 3021 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:01.560954 3021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:01.560988 3021 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:01.562609 3021 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3021
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:01.562887 3021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:01.563141 3021 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:01.568343 3029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:01.568352 3026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:01.568610 3021 server_base.cc:1034] running on GCE node
W20260620 00:18:01.568352 3027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:01.568867 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:01.569130 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:01.570300 3021 hybrid_clock.cc:648] HybridClock initialized: now 1781914681570318 us; error 59 us; skew 500 ppm
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:01.573009 3021 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:18:01.573529 3021 webserver.cc:460] Webserver started at http://127.0.98.2:39595/ using document root <none> and password file <none>
I20260620 00:18:01.573762 3021 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:01.573828 3021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:01.574003 3021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:01.575088 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "5bab63ee4d474daa844b3309cc69129e"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "3927e98988c852286639b4b698102fee"
server_key_iv: "9e9c9b896e5224de52a801050d86f931"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.575501 3021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5bab63ee4d474daa844b3309cc69129e"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "3927e98988c852286639b4b698102fee"
server_key_iv: "9e9c9b896e5224de52a801050d86f931"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.577036 3021 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:01.577914 3036 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.578130 3021 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:01.578238 3021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "5bab63ee4d474daa844b3309cc69129e"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "3927e98988c852286639b4b698102fee"
server_key_iv: "9e9c9b896e5224de52a801050d86f931"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.578333 3021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:01.594012 3021 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:01.596089 3021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:01.596287 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:01.596554 3021 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:01.596886 3021 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:01.596937 3021 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.596975 3021 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:01.597003 3021 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.603072 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44099
I20260620 00:18:01.603116 3149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44099 every 8 connection(s)
I20260620 00:18:01.603518 3021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:01.612277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 3021
I20260620 00:18:01.612360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:01.612563 392 external_mini_cluster.cc:1428] Setting key 130dc3a3a2e278024c139e9cb23a05c4
I20260620 00:18:01.613790 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:01.604291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60589 (local address 127.0.98.62:44567)
0620 00:18:01.604431 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:01.604434 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:01.604606 (+ 172us) server_negotiation.cc:407] Connection header received
0620 00:18:01.605523 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:01.605524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:01.605535 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:01.605613 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:01.606302 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.606826 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.608034 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.608287 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.608353 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:01.610684 (+ 2331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:01.610685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:01.610686 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:01.610692 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:01.612130 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:01.612642 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:01.612643 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:01.612644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:01.612704 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:01.613084 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:01.613085 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:01.613085 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:01.613244 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:01.613322 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:01.613657 (+ 335us) server_negotiation.cc:299] Negotiation successful
0620 00:18:01.613727 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 00:18:01.614136 3152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:01.604289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44567 (local address 127.0.98.2:60589)
0620 00:18:01.604453 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:01.604497 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:01.605423 (+ 926us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:01.605678 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:01.605683 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:01.605979 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:01.606180 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:01.606185 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.606944 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.606945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:01.607925 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:01.607927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.608362 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.608363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:01.608483 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:01.609043 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:01.609051 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:01.610583 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:01.612246 (+ 1663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:01.612253 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:01.612256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:01.612541 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:01.612870 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:01.612871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:01.612871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:01.613010 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:01.613325 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:01.613332 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:01.613512 (+ 180us) client_negotiation.cc:769] Sending connection context
0620 00:18:01.613638 (+ 126us) client_negotiation.cc:240] Negotiation successful
0620 00:18:01.613848 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260620 00:18:01.614640 3150 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44567
I20260620 00:18:01.614825 3150 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:01.615048 3150 heartbeater.cc:511] Master 127.0.98.62:44567 requested a full tablet report, sending...
I20260620 00:18:01.615517 2825 ts_manager.cc:194] Registered new tserver with Master: 5bab63ee4d474daa844b3309cc69129e (127.0.98.2:44099)
I20260620 00:18:01.615932 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:60589
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:18:01.639892 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44567
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:01.722988 3155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:01.723208 3155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:01.723237 3155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:01.724509 3155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:01.724557 3155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:01.724586 3155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:01.726181 3155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44567
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:01.726444 3155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:01.726677 3155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:01.731870 3161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:01.731900 3160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:01.731937 3155 server_base.cc:1034] running on GCE node
W20260620 00:18:01.731909 3163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:01.732358 3155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:01.732662 3155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:01.733809 3155 hybrid_clock.cc:648] HybridClock initialized: now 1781914681733779 us; error 47 us; skew 500 ppm
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[2780](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:01.736521 3155 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:18:01.736971 3155 webserver.cc:460] Webserver started at http://127.0.98.3:38443/ using document root <none> and password file <none>
I20260620 00:18:01.737195 3155 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:01.737262 3155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:01.737404 3155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:01.738433 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "d33af46b1f1b4048933037d307ab318c"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "b29975e918bda8251df1764901b7e28a"
server_key_iv: "fe4645d706761b8bc8f91480382609ae"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.738775 3155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d33af46b1f1b4048933037d307ab318c"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "b29975e918bda8251df1764901b7e28a"
server_key_iv: "fe4645d706761b8bc8f91480382609ae"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.740029 3155 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:18:01.740747 3170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.740914 3155 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.741006 3155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "d33af46b1f1b4048933037d307ab318c"
format_stamp: "Formatted at 2026-06-20 00:18:01 on dist-test-slave-r81b"
server_key: "b29975e918bda8251df1764901b7e28a"
server_key_iv: "fe4645d706761b8bc8f91480382609ae"
server_key_version: "encryptionkey@0"
I20260620 00:18:01.741079 3155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:01.762873 3155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:01.765316 3155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:01.765486 3155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:01.765712 3155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:01.766021 3155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:01.766074 3155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.766117 3155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:01.766144 3155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:01.771843 3155 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44831
I20260620 00:18:01.771893 3283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44831 every 8 connection(s)
I20260620 00:18:01.772261 3155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:01.774971 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 3155
I20260620 00:18:01.775043 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:01.775209 392 external_mini_cluster.cc:1428] Setting key 98b35fc33297820f37db5c632b9dc8a0
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[2780](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914681, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:01.781880 2865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:01.773012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44535 (local address 127.0.98.62:44567)
0620 00:18:01.773095 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:01.773097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:01.773257 (+ 160us) server_negotiation.cc:407] Connection header received
0620 00:18:01.774258 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:01.774260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:01.774270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:01.774323 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:01.774964 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.775663 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.776695 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.776910 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.776994 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:01.779189 (+ 2195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:01.779190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:01.779190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:01.779197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:01.780622 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:01.781025 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:01.781026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:01.781027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:01.781099 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:01.781340 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:01.781341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:01.781341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:01.781450 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:01.781504 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:01.781750 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 00:18:01.781814 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:18:01.782083 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:01.772981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44567 (local address 127.0.98.3:44535)
0620 00:18:01.773141 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:01.773177 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:01.774164 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:01.774388 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:01.774391 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:01.774650 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:01.774863 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:01.774869 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.775748 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.775750 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:01.776603 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:01.776605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:01.777004 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:01.777006 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:01.777142 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:01.777681 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:01.777689 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:01.779103 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:01.780709 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:01.780715 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:01.780717 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:01.780939 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:01.781161 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:01.781162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:01.781162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:01.781256 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:01.781498 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:01.781502 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:01.781621 (+ 119us) client_negotiation.cc:769] Sending connection context
0620 00:18:01.781733 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:18:01.781866 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":92,"threads_started":1}
I20260620 00:18:01.782563 3284 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44567
I20260620 00:18:01.782670 3284 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:01.782867 3284 heartbeater.cc:511] Master 127.0.98.62:44567 requested a full tablet report, sending...
I20260620 00:18:01.783231 2825 ts_manager.cc:194] Registered new tserver with Master: d33af46b1f1b4048933037d307ab318c (127.0.98.3:44831)
I20260620 00:18:01.783576 2825 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:44535
I20260620 00:18:01.786859 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:18:01.798177 2865 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:01.797354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59438 (local address 127.0.98.62:44567)
0620 00:18:01.797414 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:01.797416 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:01.797486 (+ 70us) server_negotiation.cc:407] Connection header received
0620 00:18:01.797538 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:01.797538 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:01.797548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:01.797616 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:01.798107 (+ 491us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59438: BlockingRecv error: recv got EOF from 127.0.0.1:59438 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:18:01.799042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 2887
I20260620 00:18:01.803504 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 3021
I20260620 00:18:01.808501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 3155
I20260620 00:18:01.813126 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 2793
2026-06-20T00:18:01Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1919 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[3296](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[3296](info): set up 2 sockets
Jun 20 00:18:01 dist-test-slave-r81b krb5kdc[3296](info): commencing operation
krb5kdc: starting...
W20260620 00:18:02.984093 392 mini_kdc.cc:121] Time spent starting KDC: real 1.150s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:03 dist-test-slave-r81b krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:18:03.111173 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37647
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37647
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:03.194805 3309 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:03.195025 3309 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:03.195048 3309 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:03.196316 3309 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:03.196364 3309 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:03.196379 3309 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:03.196393 3309 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:03.196403 3309 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:03.197913 3309 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3309
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:03.198124 3309 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:03.198313 3309 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:03.203235 3315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:03.203282 3317 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:03.203318 3309 server_base.cc:1034] running on GCE node
W20260620 00:18:03.203262 3314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:03.203675 3309 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:03.203975 3309 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:03.205137 3309 hybrid_clock.cc:648] HybridClock initialized: now 1781914683205113 us; error 35 us; skew 500 ppm
Jun 20 00:18:03 dist-test-slave-r81b krb5kdc[3296](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:03.212612 3309 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:18:03.213140 3309 webserver.cc:460] Webserver started at http://127.0.98.62:35597/ using document root <none> and password file <none>
I20260620 00:18:03.213383 3309 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:03.213451 3309 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:03.213598 3309 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:03.214726 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "bf4e8d53faef4ac4a7b5a35ef273825c"
format_stamp: "Formatted at 2026-06-20 00:18:03 on dist-test-slave-r81b"
server_key: "ba3116859c9d1c42c81afd89a53697d5"
server_key_iv: "22e7831e281a0434babe63ba2385bc2a"
server_key_version: "encryptionkey@0"
I20260620 00:18:03.215087 3309 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "bf4e8d53faef4ac4a7b5a35ef273825c"
format_stamp: "Formatted at 2026-06-20 00:18:03 on dist-test-slave-r81b"
server_key: "ba3116859c9d1c42c81afd89a53697d5"
server_key_iv: "22e7831e281a0434babe63ba2385bc2a"
server_key_version: "encryptionkey@0"
I20260620 00:18:03.216471 3309 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:03.217305 3324 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:03.217481 3309 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:03.217582 3309 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "bf4e8d53faef4ac4a7b5a35ef273825c"
format_stamp: "Formatted at 2026-06-20 00:18:03 on dist-test-slave-r81b"
server_key: "ba3116859c9d1c42c81afd89a53697d5"
server_key_iv: "22e7831e281a0434babe63ba2385bc2a"
server_key_version: "encryptionkey@0"
I20260620 00:18:03.217674 3309 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:03.228825 3309 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:03.230831 3309 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:03.230996 3309 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:03.235230 3309 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37647
I20260620 00:18:03.235265 3376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37647 every 8 connection(s)
I20260620 00:18:03.235657 3309 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:03.236197 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 3309
I20260620 00:18:03.236301 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:03.236510 392 external_mini_cluster.cc:1428] Setting key 901b3cafb6b73668e230d7a38f1cbdff
I20260620 00:18:03.236764 3377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:03.239653 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c: Bootstrap starting.
I20260620 00:18:03.240459 3377 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:03.240751 3377 log.cc:824] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:03.241474 3377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c: No bootstrap required, opened a new log
Jun 20 00:18:03 dist-test-slave-r81b krb5kdc[3296](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914683, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:03.243577 3377 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e8d53faef4ac4a7b5a35ef273825c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37647 } }
I20260620 00:18:03.243744 3377 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:03.243793 3377 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4e8d53faef4ac4a7b5a35ef273825c, State: Initialized, Role: FOLLOWER
I20260620 00:18:03.243927 3377 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e8d53faef4ac4a7b5a35ef273825c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37647 } }
I20260620 00:18:03.244024 3377 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:03.244068 3377 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:03.244117 3377 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:03.244830 3377 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e8d53faef4ac4a7b5a35ef273825c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37647 } }
I20260620 00:18:03.244946 3377 leader_election.cc:302] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf4e8d53faef4ac4a7b5a35ef273825c; no voters:
I20260620 00:18:03.245149 3377 leader_election.cc:288] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:03.245170 3382 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:03.245287 3382 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [term 1 LEADER]: Becoming Leader. State: Replica: bf4e8d53faef4ac4a7b5a35ef273825c, State: Running, Role: LEADER
I20260620 00:18:03.245455 3377 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:03.245440 3382 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e8d53faef4ac4a7b5a35ef273825c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37647 } }
I20260620 00:18:03.247392 3384 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4e8d53faef4ac4a7b5a35ef273825c. Latest consensus state: current_term: 1 leader_uuid: "bf4e8d53faef4ac4a7b5a35ef273825c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e8d53faef4ac4a7b5a35ef273825c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37647 } } }
I20260620 00:18:03.247481 3384 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:03.247812 3396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:03.247833 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:03.236951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59604 (local address 127.0.98.62:37647)
0620 00:18:03.237163 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:03.237174 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:03.237206 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:18:03.237896 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:03.237904 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:03.237957 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:03.238127 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:03.238780 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:03.239717 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:03.240536 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:03.241005 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:03.241127 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:03.243896 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:03.243901 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:03.243901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:03.243914 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:03.245886 (+ 1972us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:03.246321 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:03.246324 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:03.246325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:03.246430 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:03.246781 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:03.246782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:03.246782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:03.247131 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:03.247304 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:03.247460 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 00:18:03.247626 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
I20260620 00:18:03.247891 3383 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf4e8d53faef4ac4a7b5a35ef273825c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4e8d53faef4ac4a7b5a35ef273825c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37647 } } }
I20260620 00:18:03.247952 3383 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:03.248489 3396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:03.248910 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:18:03.249963 3396 catalog_manager.cc:1347] Generated new cluster ID: bfc5e502b98d43ca867f390d7c5f68f9
I20260620 00:18:03.250011 3396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:03.257529 3396 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:03.258293 3396 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:03.266947 3396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf4e8d53faef4ac4a7b5a35ef273825c: Generated new TSK 0
I20260620 00:18:03.267195 3396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:18:03.272444 3380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:03.268233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59606 (local address 127.0.98.62:37647)
0620 00:18:03.268281 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:03.268284 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:03.268449 (+ 165us) server_negotiation.cc:407] Connection header received
0620 00:18:03.268519 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:03.268520 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:03.268532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:03.268589 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:03.268989 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:03.269793 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:03.270614 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:03.270971 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:03.271038 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:03.271811 (+ 773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:03.271812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:03.271813 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:03.272057 (+ 244us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:03.272077 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:03.272086 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:03.272187 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:03.272323 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 00:18:03.272354 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:18:03.275199 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 3309
2026-06-20T00:18:03Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1458 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:03 dist-test-slave-r81b krb5kdc[3410](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:03 dist-test-slave-r81b krb5kdc[3410](info): set up 2 sockets
Jun 20 00:18:03 dist-test-slave-r81b krb5kdc[3410](info): commencing operation
krb5kdc: starting...
W20260620 00:18:04.324240 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:04 dist-test-slave-r81b krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:18:04.443485 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33827
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33827
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:04.526005 3423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:04.526217 3423 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:04.526242 3423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:04.527595 3423 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:04.527643 3423 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:04.527658 3423 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:04.527671 3423 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:04.527683 3423 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:04.529294 3423 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33827
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3423
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:04.529561 3423 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:04.529808 3423 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:04.535084 3429 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:04.535149 3431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:04.535084 3428 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:04.535264 3423 server_base.cc:1034] running on GCE node
I20260620 00:18:04.535617 3423 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:04.535883 3423 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:04.537084 3423 hybrid_clock.cc:648] HybridClock initialized: now 1781914684537048 us; error 50 us; skew 500 ppm
Jun 20 00:18:04 dist-test-slave-r81b krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:04.539664 3423 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:18:04.540184 3423 webserver.cc:460] Webserver started at http://127.0.98.62:42889/ using document root <none> and password file <none>
I20260620 00:18:04.540421 3423 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:04.540483 3423 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:04.540617 3423 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:04.541706 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "fb01638d08634aefafbfd84a003f47fd"
format_stamp: "Formatted at 2026-06-20 00:18:04 on dist-test-slave-r81b"
server_key: "62fa2361c71234e7816cd1cee2431d68"
server_key_iv: "602e9ada7801c95e63efa6fb9508f206"
server_key_version: "encryptionkey@0"
I20260620 00:18:04.542079 3423 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "fb01638d08634aefafbfd84a003f47fd"
format_stamp: "Formatted at 2026-06-20 00:18:04 on dist-test-slave-r81b"
server_key: "62fa2361c71234e7816cd1cee2431d68"
server_key_iv: "602e9ada7801c95e63efa6fb9508f206"
server_key_version: "encryptionkey@0"
I20260620 00:18:04.543411 3423 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:04.544226 3438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:04.544404 3423 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:04.544507 3423 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "fb01638d08634aefafbfd84a003f47fd"
format_stamp: "Formatted at 2026-06-20 00:18:04 on dist-test-slave-r81b"
server_key: "62fa2361c71234e7816cd1cee2431d68"
server_key_iv: "602e9ada7801c95e63efa6fb9508f206"
server_key_version: "encryptionkey@0"
I20260620 00:18:04.544593 3423 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:04.564855 3423 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:04.567190 3423 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:04.567379 3423 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:04.571460 3423 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33827
I20260620 00:18:04.571491 3490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33827 every 8 connection(s)
I20260620 00:18:04.571843 3423 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:04.572917 3491 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:04.575209 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd: Bootstrap starting.
I20260620 00:18:04.575773 3491 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:04.576045 3491 log.cc:824] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:04.576824 3491 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd: No bootstrap required, opened a new log
I20260620 00:18:04.578445 3491 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb01638d08634aefafbfd84a003f47fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260620 00:18:04.578588 3491 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:04.578610 3491 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb01638d08634aefafbfd84a003f47fd, State: Initialized, Role: FOLLOWER
I20260620 00:18:04.578826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 3423
I20260620 00:18:04.578809 3491 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb01638d08634aefafbfd84a003f47fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260620 00:18:04.578889 3491 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:04.578931 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:04.578945 3491 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:04.578991 3491 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:04.579140 392 external_mini_cluster.cc:1428] Setting key 48d0094bed381ecdab46fbe4c8693742
I20260620 00:18:04.580083 3491 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb01638d08634aefafbfd84a003f47fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260620 00:18:04.580231 3491 leader_election.cc:302] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb01638d08634aefafbfd84a003f47fd; no voters:
I20260620 00:18:04.580404 3491 leader_election.cc:288] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:04.580417 3496 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:04.580543 3496 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [term 1 LEADER]: Becoming Leader. State: Replica: fb01638d08634aefafbfd84a003f47fd, State: Running, Role: LEADER
I20260620 00:18:04.580626 3496 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb01638d08634aefafbfd84a003f47fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } }
I20260620 00:18:04.580678 3491 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:04.580948 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb01638d08634aefafbfd84a003f47fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb01638d08634aefafbfd84a003f47fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } } }
I20260620 00:18:04.581071 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:04.581245 3496 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb01638d08634aefafbfd84a003f47fd. Latest consensus state: current_term: 1 leader_uuid: "fb01638d08634aefafbfd84a003f47fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb01638d08634aefafbfd84a003f47fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33827 } } }
I20260620 00:18:04.581364 3496 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:04.581959 3502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:04.582553 3502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:04.584234 3502 catalog_manager.cc:1347] Generated new cluster ID: b29dd79badd94d919e87ade1968b8c31
I20260620 00:18:04.584285 3502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:18:04 dist-test-slave-r81b krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914684, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:04.589910 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:04.579667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43064 (local address 127.0.98.62:33827)
0620 00:18:04.579858 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:04.579868 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:04.579907 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:18:04.580008 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:04.580013 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:04.580058 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:04.580256 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:04.581069 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.582526 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.583368 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.583965 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.584086 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:04.586508 (+ 2422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:04.586513 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:04.586513 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:04.586525 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:04.588342 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:04.588721 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:04.588724 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:04.588724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:04.588832 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:04.589049 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:04.589049 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:04.589050 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:04.589306 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:04.589484 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:04.589560 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 00:18:04.589715 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":108,"threads_started":1}
I20260620 00:18:04.603006 3502 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:04.603564 3502 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:04.605674 3502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb01638d08634aefafbfd84a003f47fd: Generated new TSK 0
I20260620 00:18:04.605855 3502 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:18:04.638890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:04.731683 3517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:04.731885 3517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:04.732015 3517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:04.733415 3517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:04.733493 3517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:04.733523 3517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:04.735136 3517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:04.735406 3517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:04.735648 3517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:04.740859 3523 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:04.740916 3525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:04.740854 3522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:04.741084 3517 server_base.cc:1034] running on GCE node
I20260620 00:18:04.741323 3517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:04.741539 3517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:04.742694 3517 hybrid_clock.cc:648] HybridClock initialized: now 1781914684742680 us; error 29 us; skew 500 ppm
Jun 20 00:18:04 dist-test-slave-r81b krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:04.745360 3517 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:18:04.745764 3517 webserver.cc:460] Webserver started at http://127.0.98.1:35349/ using document root <none> and password file <none>
I20260620 00:18:04.745955 3517 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:04.745996 3517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:04.746100 3517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:04.747176 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "4bb75fe6a643420d9ee171146a0ba693"
format_stamp: "Formatted at 2026-06-20 00:18:04 on dist-test-slave-r81b"
server_key: "e1f751a49e187e9ed4ebaa61995b7fea"
server_key_iv: "32646a65aa88155e690aae5199a52ce4"
server_key_version: "encryptionkey@0"
I20260620 00:18:04.747579 3517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4bb75fe6a643420d9ee171146a0ba693"
format_stamp: "Formatted at 2026-06-20 00:18:04 on dist-test-slave-r81b"
server_key: "e1f751a49e187e9ed4ebaa61995b7fea"
server_key_iv: "32646a65aa88155e690aae5199a52ce4"
server_key_version: "encryptionkey@0"
I20260620 00:18:04.749120 3517 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:04.749994 3532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:04.750303 3517 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:18:04.750408 3517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "4bb75fe6a643420d9ee171146a0ba693"
format_stamp: "Formatted at 2026-06-20 00:18:04 on dist-test-slave-r81b"
server_key: "e1f751a49e187e9ed4ebaa61995b7fea"
server_key_iv: "32646a65aa88155e690aae5199a52ce4"
server_key_version: "encryptionkey@0"
I20260620 00:18:04.750483 3517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:04.768496 3517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:04.770889 3517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:04.771061 3517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:04.771306 3517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:04.771632 3517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:04.771689 3517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:04.771731 3517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:04.771762 3517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:04.777612 3517 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40935
I20260620 00:18:04.777681 3645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40935 every 8 connection(s)
I20260620 00:18:04.778026 3517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:18:04.784744 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 3517
Jun 20 00:18:04 dist-test-slave-r81b krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:04.784854 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:04.785076 392 external_mini_cluster.cc:1428] Setting key cbdd7b8eb43254b4fec1804bb37155c0
I20260620 00:18:04.788654 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:04.778810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40771 (local address 127.0.98.62:33827)
0620 00:18:04.778881 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:04.778883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:04.779107 (+ 224us) server_negotiation.cc:407] Connection header received
0620 00:18:04.779615 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:04.779616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:04.779629 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:04.779680 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:04.780914 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.781571 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.782729 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.782925 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.782990 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:04.785454 (+ 2464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:04.785456 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:04.785456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:04.785466 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:04.786987 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:04.787510 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:04.787511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:04.787512 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:04.787582 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:04.787829 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:04.787830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:04.787830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:04.787992 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:04.788072 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:04.788399 (+ 327us) server_negotiation.cc:299] Negotiation successful
0620 00:18:04.788508 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:18:04.788738 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:04.778770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33827 (local address 127.0.98.1:40771)
0620 00:18:04.778978 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:04.779018 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:04.779518 (+ 500us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:04.779747 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:04.779750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:04.780551 (+ 801us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:04.780780 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:04.780786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.781684 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.781685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:04.782647 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:04.782649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.783002 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.783003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:04.783110 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:04.783620 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:04.783628 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:04.785318 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:04.787093 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:04.787104 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:04.787107 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:04.787433 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:04.787651 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:04.787652 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:04.787652 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:04.787768 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:04.788067 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:04.788071 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:04.788262 (+ 191us) client_negotiation.cc:769] Sending connection context
0620 00:18:04.788362 (+ 100us) client_negotiation.cc:240] Negotiation successful
0620 00:18:04.788524 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":93,"threads_started":1}
I20260620 00:18:04.789323 3646 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33827
I20260620 00:18:04.789445 3646 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:04.789666 3646 heartbeater.cc:511] Master 127.0.98.62:33827 requested a full tablet report, sending...
I20260620 00:18:04.790211 3455 ts_manager.cc:194] Registered new tserver with Master: 4bb75fe6a643420d9ee171146a0ba693 (127.0.98.1:40935)
I20260620 00:18:04.790894 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40771
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:18:04.812393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:04.893950 3651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:04.894192 3651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:04.894229 3651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:04.895399 3651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:04.895468 3651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:04.895514 3651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:04.896979 3651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:04.897267 3651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:04.897557 3651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:04.904711 3657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:04.904973 3656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:04.905931 3659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:04.906548 3651 server_base.cc:1034] running on GCE node
I20260620 00:18:04.906782 3651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:04.907019 3651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:04.908210 3651 hybrid_clock.cc:648] HybridClock initialized: now 1781914684908199 us; error 37 us; skew 500 ppm
Jun 20 00:18:04 dist-test-slave-r81b krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:04.910728 3651 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:18:04.911356 3651 webserver.cc:460] Webserver started at http://127.0.98.2:45951/ using document root <none> and password file <none>
I20260620 00:18:04.911587 3651 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:04.911646 3651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:04.911742 3651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:04.912938 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "7fe0ffca9bd543ea944a940494c872c5"
format_stamp: "Formatted at 2026-06-20 00:18:04 on dist-test-slave-r81b"
server_key: "e324d6d07d10432cfe4daa022626687d"
server_key_iv: "2767de66f4f2a070767291b380c290d0"
server_key_version: "encryptionkey@0"
I20260620 00:18:04.913249 3651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7fe0ffca9bd543ea944a940494c872c5"
format_stamp: "Formatted at 2026-06-20 00:18:04 on dist-test-slave-r81b"
server_key: "e324d6d07d10432cfe4daa022626687d"
server_key_iv: "2767de66f4f2a070767291b380c290d0"
server_key_version: "encryptionkey@0"
I20260620 00:18:04.914901 3651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:04.915614 3666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:04.915781 3651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:04.915869 3651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "7fe0ffca9bd543ea944a940494c872c5"
format_stamp: "Formatted at 2026-06-20 00:18:04 on dist-test-slave-r81b"
server_key: "e324d6d07d10432cfe4daa022626687d"
server_key_iv: "2767de66f4f2a070767291b380c290d0"
server_key_version: "encryptionkey@0"
I20260620 00:18:04.915928 3651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:04.941628 3651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:04.943739 3651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:04.943850 3651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:04.944042 3651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:04.944391 3651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:04.944432 3651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:04.944476 3651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:04.944504 3651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:04.950362 3651 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35433
I20260620 00:18:04.950433 3779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35433 every 8 connection(s)
I20260620 00:18:04.950764 3651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:18:04 dist-test-slave-r81b krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:04.958022 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 3651
I20260620 00:18:04.958137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:04.958369 392 external_mini_cluster.cc:1428] Setting key c90efcfa573a6906d46780280c0c4257
I20260620 00:18:04.960657 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:04.951517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43479 (local address 127.0.98.62:33827)
0620 00:18:04.951582 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:04.951584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:04.951828 (+ 244us) server_negotiation.cc:407] Connection header received
0620 00:18:04.952413 (+ 585us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:04.952414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:04.952425 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:04.952497 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:04.953517 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.954098 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.955135 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.955338 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.955416 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:04.957571 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:04.957572 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:04.957573 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:04.957581 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:04.959265 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:04.959659 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:04.959661 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:04.959661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:04.959726 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:04.959942 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:04.959942 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:04.959943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:04.960090 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:04.960181 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:04.960470 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 00:18:04.960558 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:18:04.960865 3782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:04.951493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33827 (local address 127.0.98.2:43479)
0620 00:18:04.951660 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:04.951703 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:04.952301 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:04.952594 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:04.952598 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:04.953176 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:04.953428 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:04.953433 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.954186 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.954187 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:04.955057 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:04.955060 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:04.955427 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:04.955428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:04.955524 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:04.956071 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:04.956082 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:04.957487 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:04.959340 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:04.959345 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:04.959347 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:04.959588 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:04.959790 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:04.959790 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:04.959790 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:04.959884 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:04.960206 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:04.960211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:04.960332 (+ 121us) client_negotiation.cc:769] Sending connection context
0620 00:18:04.960464 (+ 132us) client_negotiation.cc:240] Negotiation successful
0620 00:18:04.960625 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260620 00:18:04.961476 3780 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33827
I20260620 00:18:04.961576 3780 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:04.961753 3780 heartbeater.cc:511] Master 127.0.98.62:33827 requested a full tablet report, sending...
I20260620 00:18:04.962136 3455 ts_manager.cc:194] Registered new tserver with Master: 7fe0ffca9bd543ea944a940494c872c5 (127.0.98.2:35433)
I20260620 00:18:04.962517 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43479
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:18:04.983920 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42295
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:05.066001 3785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:05.066216 3785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:05.066241 3785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:05.067330 3785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:05.067366 3785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:05.067391 3785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:05.068840 3785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42295
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33827
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3785
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:05.069029 3785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:05.069212 3785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:05.074246 3793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:05.074388 3785 server_base.cc:1034] running on GCE node
W20260620 00:18:05.074338 3790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:05.074246 3791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:05.074726 3785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:05.074985 3785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:05.076200 3785 hybrid_clock.cc:648] HybridClock initialized: now 1781914685076181 us; error 35 us; skew 500 ppm
Jun 20 00:18:05 dist-test-slave-r81b krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:05.079128 3785 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:18:05.079634 3785 webserver.cc:460] Webserver started at http://127.0.98.3:33995/ using document root <none> and password file <none>
I20260620 00:18:05.079857 3785 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:05.079905 3785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:05.080027 3785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:05.081331 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "dfca35b5b6274001887e9ecb3eb52413"
format_stamp: "Formatted at 2026-06-20 00:18:05 on dist-test-slave-r81b"
server_key: "61382172a08a44880fdd3f2cee598fe5"
server_key_iv: "fb6c621b91f8fe208d12e8638d0f28f4"
server_key_version: "encryptionkey@0"
I20260620 00:18:05.081704 3785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dfca35b5b6274001887e9ecb3eb52413"
format_stamp: "Formatted at 2026-06-20 00:18:05 on dist-test-slave-r81b"
server_key: "61382172a08a44880fdd3f2cee598fe5"
server_key_iv: "fb6c621b91f8fe208d12e8638d0f28f4"
server_key_version: "encryptionkey@0"
I20260620 00:18:05.083261 3785 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:05.084071 3800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:05.084270 3785 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:05.084365 3785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "dfca35b5b6274001887e9ecb3eb52413"
format_stamp: "Formatted at 2026-06-20 00:18:05 on dist-test-slave-r81b"
server_key: "61382172a08a44880fdd3f2cee598fe5"
server_key_iv: "fb6c621b91f8fe208d12e8638d0f28f4"
server_key_version: "encryptionkey@0"
I20260620 00:18:05.084442 3785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:05.094108 3785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:05.096468 3785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:05.096623 3785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:05.096848 3785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:05.097162 3785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:05.097214 3785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:05.097257 3785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:05.097283 3785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:05.103296 3785 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36065
I20260620 00:18:05.103431 3913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36065 every 8 connection(s)
I20260620 00:18:05.103725 3785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:05.108884 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 3785
I20260620 00:18:05.109009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:05.109257 392 external_mini_cluster.cc:1428] Setting key 4b120b588aa06ea225f71506c473a5cf
Jun 20 00:18:05 dist-test-slave-r81b krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914685, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:05.113962 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:05.104554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49323 (local address 127.0.98.62:33827)
0620 00:18:05.104628 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:05.104630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:05.104801 (+ 171us) server_negotiation.cc:407] Connection header received
0620 00:18:05.105385 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:05.105387 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:05.105397 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:05.105479 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:05.106519 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:05.107000 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:05.108099 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:05.108364 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:05.108419 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:05.111122 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:05.111124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:05.111124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:05.111131 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:05.112526 (+ 1395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:05.112941 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:05.112942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:05.112943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:05.112999 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:05.113274 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:05.113275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:05.113275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:05.113383 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:05.113437 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:05.113771 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 00:18:05.113896 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:18:05.114264 3916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:05.104512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33827 (local address 127.0.98.3:49323)
0620 00:18:05.104676 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:05.104718 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:05.105288 (+ 570us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:05.105572 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:05.105576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:05.106233 (+ 657us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:05.106434 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:05.106439 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:05.107115 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:05.107116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:05.108010 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:05.108013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:05.108446 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:05.108447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:05.108562 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:05.109226 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:05.109237 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:05.111038 (+ 1801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:05.112616 (+ 1578us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:05.112622 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:05.112624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:05.112869 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:05.113093 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:05.113094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:05.113094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:05.113211 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:05.113465 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:05.113471 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:05.113603 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 00:18:05.113714 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:18:05.113967 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260620 00:18:05.114751 3914 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33827
I20260620 00:18:05.114861 3914 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:05.115051 3914 heartbeater.cc:511] Master 127.0.98.62:33827 requested a full tablet report, sending...
I20260620 00:18:05.115427 3455 ts_manager.cc:194] Registered new tserver with Master: dfca35b5b6274001887e9ecb3eb52413 (127.0.98.3:36065)
I20260620 00:18:05.115806 3455 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:49323
I20260620 00:18:05.121446 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:18:05 dist-test-slave-r81b krb5kdc[3410](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914685, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Jun 20 00:18:05 dist-test-slave-r81b krb5kdc[3410](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914685, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:05.144507 3495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:05.137027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43074 (local address 127.0.98.62:33827)
0620 00:18:05.137105 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:05.137108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:05.137217 (+ 109us) server_negotiation.cc:407] Connection header received
0620 00:18:05.137269 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:05.137270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:05.137280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:05.137334 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:05.137975 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:05.138464 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:05.139203 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:05.139380 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:05.139430 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:05.141855 (+ 2425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:05.141857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:05.141858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:05.141867 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:05.143210 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:05.143547 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:05.143548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:05.143548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:05.143624 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:05.143822 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:05.143823 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:05.143823 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:05.144030 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:05.144123 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:05.144326 (+ 203us) server_negotiation.cc:299] Negotiation successful
0620 00:18:05.144390 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 00:18:05.144806 3455 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43074
I20260620 00:18:05.145247 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 3517
I20260620 00:18:05.149856 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 3651
I20260620 00:18:05.154042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 3785
I20260620 00:18:05.158481 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 3423
2026-06-20T00:18:05Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1885 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:05 dist-test-slave-r81b krb5kdc[3926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:05 dist-test-slave-r81b krb5kdc[3926](info): set up 2 sockets
Jun 20 00:18:05 dist-test-slave-r81b krb5kdc[3926](info): commencing operation
krb5kdc: starting...
W20260620 00:18:06.208475 392 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:06 dist-test-slave-r81b krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:18:06.333611 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42921
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42921
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:06.424772 3939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:06.424988 3939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:06.425010 3939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:06.426172 3939 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:06.426213 3939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:06.426229 3939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:06.426240 3939 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:06.426254 3939 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:06.427665 3939 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42921
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42921
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:06.427879 3939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:06.428058 3939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:06.433121 3947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:06.433133 3944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:06.433175 3945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:06.433539 3939 server_base.cc:1034] running on GCE node
I20260620 00:18:06.433852 3939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:06.434242 3939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:06.435431 3939 hybrid_clock.cc:648] HybridClock initialized: now 1781914686435390 us; error 58 us; skew 500 ppm
Jun 20 00:18:06 dist-test-slave-r81b krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:06.438190 3939 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:18:06.438674 3939 webserver.cc:460] Webserver started at http://127.0.98.62:34575/ using document root <none> and password file <none>
I20260620 00:18:06.438934 3939 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:06.439004 3939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:06.439168 3939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:06.440361 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "e41bec54b3ba42d2bc793a2a7f123f9a"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "f2e85db9abd70710ea40014ef489e94c"
server_key_iv: "04e5706f8175beb736da00697d87a304"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.440728 3939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "e41bec54b3ba42d2bc793a2a7f123f9a"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "f2e85db9abd70710ea40014ef489e94c"
server_key_iv: "04e5706f8175beb736da00697d87a304"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.442076 3939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:06.442921 3954 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.443142 3939 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:06.443250 3939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "e41bec54b3ba42d2bc793a2a7f123f9a"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "f2e85db9abd70710ea40014ef489e94c"
server_key_iv: "04e5706f8175beb736da00697d87a304"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.443346 3939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:06.462524 3939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:06.465126 3939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:06.465322 3939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:06.469295 3939 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42921
I20260620 00:18:06.469362 4006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42921 every 8 connection(s)
I20260620 00:18:06.469905 3939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:06.470593 4007 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:06.473182 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a: Bootstrap starting.
I20260620 00:18:06.473796 4007 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:06.474107 4007 log.cc:824] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:06.474869 4007 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a: No bootstrap required, opened a new log
I20260620 00:18:06.476612 4007 raft_consensus.cc:348] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41bec54b3ba42d2bc793a2a7f123f9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42921 } }
I20260620 00:18:06.476758 4007 raft_consensus.cc:374] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:06.476781 4007 raft_consensus.cc:729] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e41bec54b3ba42d2bc793a2a7f123f9a, State: Initialized, Role: FOLLOWER
I20260620 00:18:06.476953 4007 consensus_queue.cc:260] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41bec54b3ba42d2bc793a2a7f123f9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42921 } }
I20260620 00:18:06.477033 4007 raft_consensus.cc:388] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:06.477057 4007 raft_consensus.cc:482] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:06.477109 4007 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:06.478133 4007 raft_consensus.cc:504] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41bec54b3ba42d2bc793a2a7f123f9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42921 } }
I20260620 00:18:06.478224 4007 leader_election.cc:302] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e41bec54b3ba42d2bc793a2a7f123f9a; no voters:
I20260620 00:18:06.478430 4007 leader_election.cc:288] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:06.478528 4010 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:06.478719 4010 raft_consensus.cc:686] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [term 1 LEADER]: Becoming Leader. State: Replica: e41bec54b3ba42d2bc793a2a7f123f9a, State: Running, Role: LEADER
I20260620 00:18:06.478786 4007 sys_catalog.cc:564] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:06.478950 4010 consensus_queue.cc:237] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41bec54b3ba42d2bc793a2a7f123f9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42921 } }
I20260620 00:18:06.479364 4011 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e41bec54b3ba42d2bc793a2a7f123f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41bec54b3ba42d2bc793a2a7f123f9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42921 } } }
I20260620 00:18:06.479357 4012 sys_catalog.cc:455] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [sys.catalog]: SysCatalogTable state changed. Reason: New leader e41bec54b3ba42d2bc793a2a7f123f9a. Latest consensus state: current_term: 1 leader_uuid: "e41bec54b3ba42d2bc793a2a7f123f9a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e41bec54b3ba42d2bc793a2a7f123f9a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42921 } } }
I20260620 00:18:06.479460 4011 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:06.479465 4012 sys_catalog.cc:458] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:06.479691 4016 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:06.479820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 3939
I20260620 00:18:06.479918 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:06.480114 392 external_mini_cluster.cc:1428] Setting key d8c2779381fd2d3ac06a2b64dea3c366
I20260620 00:18:06.480331 4016 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:06.482048 4016 catalog_manager.cc:1347] Generated new cluster ID: 8338a68b9ee2450aa2c96237bc66e768
I20260620 00:18:06.482101 4016 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:18:06 dist-test-slave-r81b krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914686, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:06.490338 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:06.480804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.0.98.62:42921)
0620 00:18:06.480980 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:06.480989 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:06.481029 (+ 40us) server_negotiation.cc:407] Connection header received
0620 00:18:06.481165 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:06.481172 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:06.481238 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:06.482179 (+ 941us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:06.482278 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.483181 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.483946 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.484505 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.484630 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:06.486963 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:06.486968 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:06.486969 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:06.486981 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:06.488750 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:06.489160 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:06.489163 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:06.489164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:06.489261 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:06.489476 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:06.489477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:06.489477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:06.489729 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:06.489911 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:06.490011 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:18:06.490152 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":104,"threads_started":1}
I20260620 00:18:06.498957 4016 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:06.499493 4016 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:06.505128 4016 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e41bec54b3ba42d2bc793a2a7f123f9a: Generated new TSK 0
I20260620 00:18:06.505333 4016 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:18:06.540235 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42921
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:06.626407 4033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:06.626617 4033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:06.626641 4033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:06.627827 4033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:06.627873 4033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:06.627900 4033 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:06.629381 4033 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:06.629638 4033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:06.629911 4033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:06.635453 4039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:06.635442 4041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:06.635633 4033 server_base.cc:1034] running on GCE node
W20260620 00:18:06.635442 4038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:06.635972 4033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:06.636288 4033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:06.637481 4033 hybrid_clock.cc:648] HybridClock initialized: now 1781914686637459 us; error 44 us; skew 500 ppm
Jun 20 00:18:06 dist-test-slave-r81b krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:06.640244 4033 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:18:06.640738 4033 webserver.cc:460] Webserver started at http://127.0.98.1:33911/ using document root <none> and password file <none>
I20260620 00:18:06.640928 4033 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:06.640969 4033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:06.641062 4033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:06.642082 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "cc275192702842eb8e166b106cf1f005"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "ae746551c7e950bc08dd84ea8e958753"
server_key_iv: "0c538e412440252aed5623b12c78d876"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.642385 4033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cc275192702842eb8e166b106cf1f005"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "ae746551c7e950bc08dd84ea8e958753"
server_key_iv: "0c538e412440252aed5623b12c78d876"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.643693 4033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:06.644621 4048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.644879 4033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:18:06.644981 4033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "cc275192702842eb8e166b106cf1f005"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "ae746551c7e950bc08dd84ea8e958753"
server_key_iv: "0c538e412440252aed5623b12c78d876"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.645048 4033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:06.665920 4033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:06.668372 4033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:06.668497 4033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:06.668690 4033 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:06.669063 4033 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:06.669102 4033 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.669126 4033 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:06.669173 4033 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.674974 4033 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38947
I20260620 00:18:06.675084 4161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38947 every 8 connection(s)
I20260620 00:18:06.675384 4033 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:18:06.675812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 4033
I20260620 00:18:06.675921 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:06.676116 392 external_mini_cluster.cc:1428] Setting key 845e4f7bedc37a9622f7aec0a4bfad79
Jun 20 00:18:06 dist-test-slave-r81b krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:18:06.686836 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:06.676782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54547 (local address 127.0.98.62:42921)
0620 00:18:06.676837 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:06.676839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:06.677083 (+ 244us) server_negotiation.cc:407] Connection header received
0620 00:18:06.678107 (+ 1024us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:06.678109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:06.678125 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:06.678174 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:06.678853 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.679629 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.680942 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.681171 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.681260 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:06.683868 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:06.683870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:06.683870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:06.683879 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:06.685369 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:06.685817 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:06.685818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:06.685818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:06.685880 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:06.686146 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:06.686147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:06.686147 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:06.686274 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:06.686348 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:06.686639 (+ 291us) server_negotiation.cc:299] Negotiation successful
0620 00:18:06.686735 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:18:06.687038 4164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:06.676787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42921 (local address 127.0.98.1:54547)
0620 00:18:06.676975 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:06.677012 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:06.678001 (+ 989us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:06.678251 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:06.678257 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:06.678563 (+ 306us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:06.678760 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:06.678764 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.679727 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.679729 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:06.680841 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:06.680844 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.681278 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.681279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:06.681423 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:06.682027 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:06.682035 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:06.683677 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:06.685469 (+ 1792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:06.685476 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:06.685479 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:06.685712 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:06.685948 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:06.685949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:06.685949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:06.686077 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:06.686348 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:06.686355 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:06.686506 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 00:18:06.686617 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:18:06.686794 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":119,"threads_started":1}
I20260620 00:18:06.687625 4162 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42921
I20260620 00:18:06.687740 4162 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:06.687927 4162 heartbeater.cc:511] Master 127.0.98.62:42921 requested a full tablet report, sending...
I20260620 00:18:06.688539 3971 ts_manager.cc:194] Registered new tserver with Master: cc275192702842eb8e166b106cf1f005 (127.0.98.1:38947)
I20260620 00:18:06.689270 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54547
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:18:06.703197 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42921
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:06.786707 4167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:06.786942 4167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:06.786967 4167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:06.788318 4167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:06.788365 4167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:06.788395 4167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:06.789906 4167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4167
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:06.790165 4167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:06.790418 4167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:06.795487 4173 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:06.795663 4167 server_base.cc:1034] running on GCE node
W20260620 00:18:06.795487 4172 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:06.795486 4175 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:06.796016 4167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:06.796320 4167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:06.797542 4167 hybrid_clock.cc:648] HybridClock initialized: now 1781914686797528 us; error 31 us; skew 500 ppm
Jun 20 00:18:06 dist-test-slave-r81b krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:06.800354 4167 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:18:06.800864 4167 webserver.cc:460] Webserver started at http://127.0.98.2:41831/ using document root <none> and password file <none>
I20260620 00:18:06.801102 4167 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:06.801173 4167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:06.801333 4167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:06.802466 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "9fa2609ff151401892805644e88f6e50"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "a77db22ac95cc8502e3f2ac66c0cc422"
server_key_iv: "1a95614db5c102f623dc5303d17ed474"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.802845 4167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "9fa2609ff151401892805644e88f6e50"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "a77db22ac95cc8502e3f2ac66c0cc422"
server_key_iv: "1a95614db5c102f623dc5303d17ed474"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.804265 4167 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 00:18:06.805083 4182 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.805425 4167 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:06.805532 4167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "9fa2609ff151401892805644e88f6e50"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "a77db22ac95cc8502e3f2ac66c0cc422"
server_key_iv: "1a95614db5c102f623dc5303d17ed474"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.805624 4167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:06.821794 4167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:06.825815 4167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:06.826023 4167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:06.826333 4167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:06.826709 4167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:06.826766 4167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.826812 4167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:06.826867 4167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.834323 4167 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45061
I20260620 00:18:06.834359 4295 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45061 every 8 connection(s)
I20260620 00:18:06.834724 4167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:06.838641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 4167
I20260620 00:18:06.838801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:06.839006 392 external_mini_cluster.cc:1428] Setting key 8d579800e376e27a041500ec4626ee08
Jun 20 00:18:06 dist-test-slave-r81b krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:06.847702 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:06.835581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49595 (local address 127.0.98.62:42921)
0620 00:18:06.835655 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:06.835658 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:06.835907 (+ 249us) server_negotiation.cc:407] Connection header received
0620 00:18:06.837076 (+ 1169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:06.837078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:06.837090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:06.837154 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:06.837884 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.838535 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.840171 (+ 1636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.840501 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.840582 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:06.843276 (+ 2694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:06.843278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:06.843279 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:06.843288 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:06.845227 (+ 1939us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:06.845835 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:06.845837 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:06.845838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:06.846054 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:06.846583 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:06.846585 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:06.846585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:06.846755 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:06.846872 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:06.847471 (+ 599us) server_negotiation.cc:299] Negotiation successful
0620 00:18:06.847599 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:18:06.847896 4298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:06.835540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42921 (local address 127.0.98.2:49595)
0620 00:18:06.835771 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:06.835816 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:06.836946 (+ 1130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:06.837221 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:06.837224 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:06.837533 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:06.837764 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:06.837769 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.838828 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.838830 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:06.840045 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:06.840048 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.840609 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.840610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:06.840790 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:06.841491 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:06.841501 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:06.843158 (+ 1657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:06.845340 (+ 2182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:06.845349 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:06.845352 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:06.845689 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:06.846213 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:06.846215 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:06.846215 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:06.846482 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:06.846857 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:06.846869 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:06.847264 (+ 395us) client_negotiation.cc:769] Sending connection context
0620 00:18:06.847413 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 00:18:06.847611 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:18:06.848732 4296 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42921
I20260620 00:18:06.848862 4296 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:06.849107 4296 heartbeater.cc:511] Master 127.0.98.62:42921 requested a full tablet report, sending...
I20260620 00:18:06.849572 3971 ts_manager.cc:194] Registered new tserver with Master: 9fa2609ff151401892805644e88f6e50 (127.0.98.2:45061)
I20260620 00:18:06.850127 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:18:06.869475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42921
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:06.955139 4301 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:06.955355 4301 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:06.955381 4301 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:06.956594 4301 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:06.956640 4301 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:06.956667 4301 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:06.958137 4301 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42921
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4301
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:06.958338 4301 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:06.958518 4301 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:06.963687 4306 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:06.963690 4309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:06.963915 4301 server_base.cc:1034] running on GCE node
W20260620 00:18:06.963784 4307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:06.964354 4301 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:06.964637 4301 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:06.965830 4301 hybrid_clock.cc:648] HybridClock initialized: now 1781914686965817 us; error 31 us; skew 500 ppm
Jun 20 00:18:06 dist-test-slave-r81b krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:06.968515 4301 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:18:06.968991 4301 webserver.cc:460] Webserver started at http://127.0.98.3:33675/ using document root <none> and password file <none>
I20260620 00:18:06.969267 4301 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:06.969352 4301 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:06.969511 4301 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:06.970719 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "a70fa8f6dccb4e76865259998c937adf"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "68b3b9d3288fe99e85f48a9da75272b0"
server_key_iv: "27ae175f168ce12b9c05ca817ce2a0e2"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.971112 4301 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a70fa8f6dccb4e76865259998c937adf"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "68b3b9d3288fe99e85f48a9da75272b0"
server_key_iv: "27ae175f168ce12b9c05ca817ce2a0e2"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.972659 4301 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:06.973589 4316 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.973815 4301 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:06.973932 4301 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "a70fa8f6dccb4e76865259998c937adf"
format_stamp: "Formatted at 2026-06-20 00:18:06 on dist-test-slave-r81b"
server_key: "68b3b9d3288fe99e85f48a9da75272b0"
server_key_iv: "27ae175f168ce12b9c05ca817ce2a0e2"
server_key_version: "encryptionkey@0"
I20260620 00:18:06.974014 4301 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:06.982419 4301 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:06.984587 4301 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:06.984779 4301 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:06.985033 4301 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:06.985347 4301 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:06.985405 4301 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.985453 4301 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:06.985488 4301 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:06.991976 4301 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36887
I20260620 00:18:06.992048 4429 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36887 every 8 connection(s)
I20260620 00:18:06.992453 4301 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:06.994575 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 4301
I20260620 00:18:06.994704 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:06.994925 392 external_mini_cluster.cc:1428] Setting key 429993f902a5c3b4afdea0b78d78589a
Jun 20 00:18:06 dist-test-slave-r81b krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914686, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:07.003018 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:06.993358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47649 (local address 127.0.98.62:42921)
0620 00:18:06.993427 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:06.993429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:06.993499 (+ 70us) server_negotiation.cc:407] Connection header received
0620 00:18:06.994532 (+ 1033us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:06.994534 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:06.994548 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:06.994610 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:06.995434 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.996098 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.997250 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.997483 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.997577 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:06.999994 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:06.999995 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:06.999996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:07.000004 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:07.001541 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:07.002020 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:07.002021 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:07.002021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:07.002084 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:07.002353 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:07.002355 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:07.002355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:07.002482 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:07.002556 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:07.002840 (+ 284us) server_negotiation.cc:299] Negotiation successful
0620 00:18:07.002922 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:18:07.003243 4432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:06.993208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42921 (local address 127.0.98.3:47649)
0620 00:18:06.993377 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:06.993424 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:06.994414 (+ 990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:06.994701 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:06.994706 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:06.995069 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:06.995323 (+ 254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:06.995330 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.996242 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.996243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:06.997149 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:06.997151 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:06.997588 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:06.997590 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:06.997735 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:06.998294 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:06.998302 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:06.999888 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:07.001657 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:07.001663 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:07.001666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:07.001927 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:07.002157 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:07.002158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:07.002158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:07.002278 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:07.002556 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:07.002563 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:07.002705 (+ 142us) client_negotiation.cc:769] Sending connection context
0620 00:18:07.002819 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:18:07.002980 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260620 00:18:07.003841 4430 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42921
I20260620 00:18:07.003940 4430 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:07.004238 4430 heartbeater.cc:511] Master 127.0.98.62:42921 requested a full tablet report, sending...
I20260620 00:18:07.004671 3971 ts_manager.cc:194] Registered new tserver with Master: a70fa8f6dccb4e76865259998c937adf (127.0.98.3:36887)
I20260620 00:18:07.005123 3971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47649
I20260620 00:18:07.007010 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Jun 20 00:18:07 dist-test-slave-r81b krb5kdc[3926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:18:07 dist-test-slave-r81b krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:18:07.032485 4437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:07.023206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56046 (local address 127.0.98.1:38947)
0620 00:18:07.023382 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:07.023398 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:07.023413 (+ 15us) server_negotiation.cc:407] Connection header received
0620 00:18:07.023440 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:07.023442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:07.023456 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:07.023534 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:07.024270 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:07.025078 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:07.025957 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:07.026290 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:07.026370 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:07.028972 (+ 2602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:07.028981 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:07.028981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:07.028989 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:07.030989 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:07.031428 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:07.031432 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:07.031433 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:07.031520 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:07.031766 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:07.031768 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:07.031768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:07.031936 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:07.032097 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:07.032251 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 00:18:07.032344 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
Jun 20 00:18:07 dist-test-slave-r81b krb5kdc[3926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914687, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:07.041246 4029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:07.034001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33156 (local address 127.0.98.62:42921)
0620 00:18:07.034070 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:07.034071 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:07.034082 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:07.034141 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:07.034142 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:07.034151 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:07.034209 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:07.034676 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:07.035159 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:07.035843 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:07.036036 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:07.036108 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:07.038478 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:07.038480 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:07.038480 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:07.038487 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:07.040057 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:07.040451 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:07.040452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:07.040452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:07.040505 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:07.040758 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:07.040758 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:07.040759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:07.040899 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:07.040974 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:07.041074 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:18:07.041140 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 00:18:07.041532 3971 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33156
I20260620 00:18:07.042093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 4033
I20260620 00:18:07.047626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 4167
I20260620 00:18:07.051918 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 4301
I20260620 00:18:07.056269 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 3939
2026-06-20T00:18:07Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1897 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:07 dist-test-slave-r81b krb5kdc[4442](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:07 dist-test-slave-r81b krb5kdc[4442](info): set up 2 sockets
Jun 20 00:18:07 dist-test-slave-r81b krb5kdc[4442](info): commencing operation
krb5kdc: starting...
W20260620 00:18:08.162773 392 mini_kdc.cc:121] Time spent starting KDC: real 1.086s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260620 00:18:08.424180 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.163s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:08 dist-test-slave-r81b krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:18:08.527806 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34645
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34645
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:08.609472 4455 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:08.609704 4455 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:08.609728 4455 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:08.610875 4455 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:08.610913 4455 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:08.610926 4455 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:08.610939 4455 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:08.610951 4455 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:08.612536 4455 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4455
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:08.612743 4455 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:08.612921 4455 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:08.617756 4463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:08.617769 4460 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:08.618031 4455 server_base.cc:1034] running on GCE node
W20260620 00:18:08.617769 4461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:08.618450 4455 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:08.618821 4455 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:08.619979 4455 hybrid_clock.cc:648] HybridClock initialized: now 1781914688619949 us; error 45 us; skew 500 ppm
Jun 20 00:18:08 dist-test-slave-r81b krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:08.626030 4455 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:18:08.626421 4455 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:08.626477 4455 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:08.626608 4455 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:08.627700 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "67eb9463ee8e4e14b30983f8f4891a73"
format_stamp: "Formatted at 2026-06-20 00:18:08 on dist-test-slave-r81b"
server_key: "cb95047be0e50d25b41034461699a67f"
server_key_iv: "e0f32593b2bdd6e5fdfb2377d3a3f217"
server_key_version: "encryptionkey@0"
I20260620 00:18:08.628118 4455 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "67eb9463ee8e4e14b30983f8f4891a73"
format_stamp: "Formatted at 2026-06-20 00:18:08 on dist-test-slave-r81b"
server_key: "cb95047be0e50d25b41034461699a67f"
server_key_iv: "e0f32593b2bdd6e5fdfb2377d3a3f217"
server_key_version: "encryptionkey@0"
I20260620 00:18:08.629546 4455 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:08.630378 4469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:08.630581 4455 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:08.630672 4455 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "67eb9463ee8e4e14b30983f8f4891a73"
format_stamp: "Formatted at 2026-06-20 00:18:08 on dist-test-slave-r81b"
server_key: "cb95047be0e50d25b41034461699a67f"
server_key_iv: "e0f32593b2bdd6e5fdfb2377d3a3f217"
server_key_version: "encryptionkey@0"
I20260620 00:18:08.630774 4455 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:08.647778 4455 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:08.650158 4455 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:08.650421 4455 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:08.654840 4455 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34645
I20260620 00:18:08.654865 4521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34645 every 8 connection(s)
I20260620 00:18:08.655159 4455 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:08.656299 4522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:08.658507 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73: Bootstrap starting.
I20260620 00:18:08.659070 4522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:08.659304 4522 log.cc:824] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:08.659960 4522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73: No bootstrap required, opened a new log
I20260620 00:18:08.661525 4522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67eb9463ee8e4e14b30983f8f4891a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } }
I20260620 00:18:08.661706 4522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:08.661732 4522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67eb9463ee8e4e14b30983f8f4891a73, State: Initialized, Role: FOLLOWER
I20260620 00:18:08.661877 4522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67eb9463ee8e4e14b30983f8f4891a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } }
I20260620 00:18:08.661968 4522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:08.661990 4522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:08.662019 4522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:08.662815 4522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67eb9463ee8e4e14b30983f8f4891a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } }
I20260620 00:18:08.662933 4522 leader_election.cc:302] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67eb9463ee8e4e14b30983f8f4891a73; no voters:
I20260620 00:18:08.662982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 4455
I20260620 00:18:08.663072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:08.663111 4522 leader_election.cc:288] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:08.663173 4525 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:08.663285 392 external_mini_cluster.cc:1428] Setting key e1bf2e51cacf270f9e3a1e6c3cb38c55
I20260620 00:18:08.663374 4522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:08.663398 4525 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [term 1 LEADER]: Becoming Leader. State: Replica: 67eb9463ee8e4e14b30983f8f4891a73, State: Running, Role: LEADER
I20260620 00:18:08.663509 4525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67eb9463ee8e4e14b30983f8f4891a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } }
I20260620 00:18:08.663866 4526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67eb9463ee8e4e14b30983f8f4891a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67eb9463ee8e4e14b30983f8f4891a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } } }
I20260620 00:18:08.663946 4526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:08.664018 4527 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67eb9463ee8e4e14b30983f8f4891a73. Latest consensus state: current_term: 1 leader_uuid: "67eb9463ee8e4e14b30983f8f4891a73" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67eb9463ee8e4e14b30983f8f4891a73" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34645 } } }
I20260620 00:18:08.664075 4527 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:08.664319 4535 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:08.664852 4535 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:08.666610 4535 catalog_manager.cc:1347] Generated new cluster ID: b697c9882bfc41daaf0bb00978db34c0
I20260620 00:18:08.666666 4535 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:18:08 dist-test-slave-r81b krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914688, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:08.674476 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:08.663877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.0.98.62:34645)
0620 00:18:08.664216 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:08.664225 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:08.664260 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:18:08.664356 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:08.664362 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:08.664438 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:08.664607 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:08.665471 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:08.666912 (+ 1441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:08.667673 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:08.668114 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:08.668277 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:08.670583 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:08.670588 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:08.670588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:08.670607 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:08.672889 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:08.673279 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:08.673282 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:08.673283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:08.673379 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:08.673588 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:08.673589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:08.673590 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:08.673846 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:08.674035 (+ 189us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:08.674105 (+ 70us) server_negotiation.cc:299] Negotiation successful
0620 00:18:08.674253 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":293,"thread_start_us":71,"threads_started":1}
I20260620 00:18:08.676306 4535 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:08.676862 4535 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:08.679662 4535 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73: Generated new TSK 0
I20260620 00:18:08.679889 4535 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:18:08.715217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:08.798959 4548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:08.799196 4548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:08.799221 4548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:08.800482 4548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:08.800526 4548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:08.800555 4548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:08.802126 4548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:08.802388 4548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:08.802644 4548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:08.807627 4554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:08.807670 4553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:08.807704 4556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:08.808208 4548 server_base.cc:1034] running on GCE node
I20260620 00:18:08.808461 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:08.808717 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:08.809890 4548 hybrid_clock.cc:648] HybridClock initialized: now 1781914688809881 us; error 34 us; skew 500 ppm
Jun 20 00:18:08 dist-test-slave-r81b krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:08.812948 4548 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:18:08.813393 4548 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:08.813469 4548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:08.813617 4548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:08.814774 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "a126afe53fb74119adb91b63106d69e5"
format_stamp: "Formatted at 2026-06-20 00:18:08 on dist-test-slave-r81b"
server_key: "371494ef2804a363a7c4164cb850d1f7"
server_key_iv: "d179267d5f1dfd72d96617e3712911eb"
server_key_version: "encryptionkey@0"
I20260620 00:18:08.815172 4548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a126afe53fb74119adb91b63106d69e5"
format_stamp: "Formatted at 2026-06-20 00:18:08 on dist-test-slave-r81b"
server_key: "371494ef2804a363a7c4164cb850d1f7"
server_key_iv: "d179267d5f1dfd72d96617e3712911eb"
server_key_version: "encryptionkey@0"
I20260620 00:18:08.816661 4548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:08.817672 4562 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:08.817930 4548 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:08.818046 4548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "a126afe53fb74119adb91b63106d69e5"
format_stamp: "Formatted at 2026-06-20 00:18:08 on dist-test-slave-r81b"
server_key: "371494ef2804a363a7c4164cb850d1f7"
server_key_iv: "d179267d5f1dfd72d96617e3712911eb"
server_key_version: "encryptionkey@0"
I20260620 00:18:08.818143 4548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:08.831722 4548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:08.834156 4548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:08.834321 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:08.834616 4548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:08.834939 4548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:08.834991 4548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:08.835032 4548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:08.835057 4548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:08.841264 4548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42251
I20260620 00:18:08.841346 4675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42251 every 8 connection(s)
I20260620 00:18:08.841699 4548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:18:08 dist-test-slave-r81b krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:08.850803 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 4548
I20260620 00:18:08.850946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:08.851186 392 external_mini_cluster.cc:1428] Setting key 1d3ebec5022e89498dee3c66927afbdd
I20260620 00:18:08.852119 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:08.842499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48255 (local address 127.0.98.62:34645)
0620 00:18:08.842545 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:08.842547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:08.842793 (+ 246us) server_negotiation.cc:407] Connection header received
0620 00:18:08.843331 (+ 538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:08.843333 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:08.843348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:08.843409 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:08.844445 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:08.844967 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:08.846043 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:08.846263 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:08.846352 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:08.848900 (+ 2548us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:08.848901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:08.848902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:08.848910 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:08.850417 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:08.850978 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:08.850980 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:08.850980 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:08.851068 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:08.851402 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:08.851403 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:08.851404 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:08.851574 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:08.851641 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:08.851949 (+ 308us) server_negotiation.cc:299] Negotiation successful
0620 00:18:08.852052 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:18:08.852451 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:08.842459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34645 (local address 127.0.98.1:48255)
0620 00:18:08.842662 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:08.842701 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:08.843221 (+ 520us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:08.843485 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:08.843489 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:08.843781 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:08.844329 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:08.844336 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:08.845095 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:08.845096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:08.845935 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:08.845938 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:08.846365 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:08.846366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:08.846506 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:08.847063 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:08.847072 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:08.848778 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:08.850534 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:08.850543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:08.850545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:08.850874 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:08.851148 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:08.851149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:08.851150 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:08.851300 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:08.851633 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:08.851640 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:08.851812 (+ 172us) client_negotiation.cc:769] Sending connection context
0620 00:18:08.851916 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 00:18:08.852122 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":131,"threads_started":1}
I20260620 00:18:08.853057 4676 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34645
I20260620 00:18:08.853195 4676 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:08.853385 4676 heartbeater.cc:511] Master 127.0.98.62:34645 requested a full tablet report, sending...
I20260620 00:18:08.854084 4485 ts_manager.cc:194] Registered new tserver with Master: a126afe53fb74119adb91b63106d69e5 (127.0.98.1:42251)
I20260620 00:18:08.854907 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48255
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:18:08.880610 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:08.966756 4681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:08.966974 4681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:08.967000 4681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:08.968219 4681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:08.968264 4681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:08.968292 4681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:08.969923 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:08.970177 4681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:08.970412 4681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:08.975503 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:08.975505 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:08.975505 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:08.975847 4681 server_base.cc:1034] running on GCE node
I20260620 00:18:08.976105 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:08.976415 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:08.977564 4681 hybrid_clock.cc:648] HybridClock initialized: now 1781914688977536 us; error 42 us; skew 500 ppm
Jun 20 00:18:08 dist-test-slave-r81b krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:08.980298 4681 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:18:08.980679 4681 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:08.980741 4681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:08.980896 4681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:08.981993 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "41d00bcb99cf46348a0c97fdaeabef19"
format_stamp: "Formatted at 2026-06-20 00:18:08 on dist-test-slave-r81b"
server_key: "7e05e24ab4dae0407024953ecbe0062e"
server_key_iv: "6b10a7c5a496222f507b924ac8793947"
server_key_version: "encryptionkey@0"
I20260620 00:18:08.982370 4681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "41d00bcb99cf46348a0c97fdaeabef19"
format_stamp: "Formatted at 2026-06-20 00:18:08 on dist-test-slave-r81b"
server_key: "7e05e24ab4dae0407024953ecbe0062e"
server_key_iv: "6b10a7c5a496222f507b924ac8793947"
server_key_version: "encryptionkey@0"
I20260620 00:18:08.983786 4681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:08.984675 4695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:08.984875 4681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:18:08.984967 4681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "41d00bcb99cf46348a0c97fdaeabef19"
format_stamp: "Formatted at 2026-06-20 00:18:08 on dist-test-slave-r81b"
server_key: "7e05e24ab4dae0407024953ecbe0062e"
server_key_iv: "6b10a7c5a496222f507b924ac8793947"
server_key_version: "encryptionkey@0"
I20260620 00:18:08.985044 4681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:08.992949 4681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:08.994932 4681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:08.995080 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:08.995340 4681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:08.995669 4681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:08.995728 4681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:08.995771 4681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:08.995800 4681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.001791 4681 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33767
I20260620 00:18:09.001858 4808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33767 every 8 connection(s)
I20260620 00:18:09.002152 4681 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:09.005921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 4681
I20260620 00:18:09.006026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:09.006219 392 external_mini_cluster.cc:1428] Setting key 542fc8609ef0ca6a5a0ebf14e1ca2c04
Jun 20 00:18:09 dist-test-slave-r81b krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914688, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:09.013010 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.003039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53485 (local address 127.0.98.62:34645)
0620 00:18:09.003091 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.003093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.003231 (+ 138us) server_negotiation.cc:407] Connection header received
0620 00:18:09.003761 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.003763 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.003774 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.003838 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.004919 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.005424 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.006860 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.007101 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.007155 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.010002 (+ 2847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.010004 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.010004 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:09.010013 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.011451 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:09.011918 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:09.011919 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:09.011920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:09.011985 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:09.012320 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:09.012321 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:09.012321 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:09.012449 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.012526 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.012819 (+ 293us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.012912 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:18:09.013240 4811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.002911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34645 (local address 127.0.98.2:53485)
0620 00:18:09.003089 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.003132 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.003633 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.003930 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.003934 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.004320 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:09.004826 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.004831 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.005505 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.005506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.006763 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.006765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.007182 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.007183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.007380 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.007936 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:09.007947 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:09.009881 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:09.011562 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:09.011570 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:09.011574 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:09.011828 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:09.012053 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:09.012054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:09.012054 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:09.012250 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:09.012517 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:09.012523 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:09.012679 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.012789 (+ 110us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.012961 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":102,"threads_started":1}
I20260620 00:18:09.013710 4809 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34645
I20260620 00:18:09.013808 4809 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:09.013988 4809 heartbeater.cc:511] Master 127.0.98.62:34645 requested a full tablet report, sending...
I20260620 00:18:09.014396 4485 ts_manager.cc:194] Registered new tserver with Master: 41d00bcb99cf46348a0c97fdaeabef19 (127.0.98.2:33767)
I20260620 00:18:09.014796 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53485
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:18:09.035889 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35063
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:09.119691 4814 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:09.120002 4814 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:09.120048 4814 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:09.121466 4814 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:09.121548 4814 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:09.121595 4814 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:09.123301 4814 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35063
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4814
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:09.123613 4814 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:09.123872 4814 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:09.129233 4820 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.129220 4819 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.129221 4822 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:09.129385 4814 server_base.cc:1034] running on GCE node
I20260620 00:18:09.129745 4814 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:09.130048 4814 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:09.131232 4814 hybrid_clock.cc:648] HybridClock initialized: now 1781914689131221 us; error 54 us; skew 500 ppm
Jun 20 00:18:09 dist-test-slave-r81b krb5kdc[4442](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:09.134320 4814 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:18:09.134845 4814 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:09.134902 4814 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:09.135039 4814 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:09.136375 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "3b4bcbefb8f44edeb9d76d3e2064266d"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "8a298e2f0de44f15d3085848ecda9cce"
server_key_iv: "06b928612748741dfbd87501e95f593f"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.136792 4814 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3b4bcbefb8f44edeb9d76d3e2064266d"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "8a298e2f0de44f15d3085848ecda9cce"
server_key_iv: "06b928612748741dfbd87501e95f593f"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.138228 4814 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:09.139168 4828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.139367 4814 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:09.139464 4814 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "3b4bcbefb8f44edeb9d76d3e2064266d"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "8a298e2f0de44f15d3085848ecda9cce"
server_key_iv: "06b928612748741dfbd87501e95f593f"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.139530 4814 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:09.154597 4814 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:09.156982 4814 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:09.157106 4814 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:09.157308 4814 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:09.157639 4814 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:09.157676 4814 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.157724 4814 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:09.157752 4814 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.163659 4814 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33827
I20260620 00:18:09.163717 4941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33827 every 8 connection(s)
I20260620 00:18:09.164017 4814 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:18:09 dist-test-slave-r81b krb5kdc[4442](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914689, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:09.171588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 4814
I20260620 00:18:09.171679 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:09.171927 392 external_mini_cluster.cc:1428] Setting key a003a40527ce653ff9227262c6f0b6e4
I20260620 00:18:09.174476 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.164797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57585 (local address 127.0.98.62:34645)
0620 00:18:09.164875 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.164878 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.165240 (+ 362us) server_negotiation.cc:407] Connection header received
0620 00:18:09.165773 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.165775 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.165784 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.165846 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.166788 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.167332 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.168524 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.168732 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.168800 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.171070 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.171072 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.171072 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:09.171080 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.173022 (+ 1942us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:09.173443 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:09.173444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:09.173444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:09.173515 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:09.173793 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:09.173794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:09.173795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:09.173921 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.173998 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.174297 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.174382 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:18:09.174718 4944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.164792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34645 (local address 127.0.98.3:57585)
0620 00:18:09.165136 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.165173 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.165677 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.165902 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.165906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.166177 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:09.166672 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.166677 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.167445 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.167446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.168425 (+ 979us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.168428 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.168813 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.168813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.168911 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.169444 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:09.169481 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:09.170965 (+ 1484us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:09.173110 (+ 2145us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:09.173118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:09.173120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:09.173357 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:09.173580 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:09.173581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:09.173581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:09.173723 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:09.173988 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:09.173994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:09.174150 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.174266 (+ 116us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.174462 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":266,"threads_started":1}
I20260620 00:18:09.175143 4942 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34645
I20260620 00:18:09.175228 4942 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:09.175387 4942 heartbeater.cc:511] Master 127.0.98.62:34645 requested a full tablet report, sending...
I20260620 00:18:09.175794 4485 ts_manager.cc:194] Registered new tserver with Master: 3b4bcbefb8f44edeb9d76d3e2064266d (127.0.98.3:33827)
I20260620 00:18:09.176282 4485 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57585
I20260620 00:18:09.184067 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:18:09.192070 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.185488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.0.98.62:34645)
0620 00:18:09.185594 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.185596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.185755 (+ 159us) server_negotiation.cc:407] Connection header received
0620 00:18:09.185786 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.185786 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.185801 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.185863 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.186454 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.187004 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.187760 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.188024 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.188084 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.189640 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.189642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.189642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:09.189648 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.191047 (+ 1399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:09.191367 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:09.191368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:09.191368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:09.191420 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:09.191594 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:09.191594 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:09.191594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:09.191765 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.191838 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.191940 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.191999 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 00:18:09.193926 4485 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:18:09.195436 4485 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:09.202126 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.198099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40414 (local address 127.0.98.3:33827)
0620 00:18:09.198273 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.198277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.198459 (+ 182us) server_negotiation.cc:407] Connection header received
0620 00:18:09.198501 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.198503 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.198519 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.198592 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:09.198813 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.199949 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.201200 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.201689 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.201786 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.201844 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.201996 (+ 152us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.202036 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":78,"threads_started":1}
I20260620 00:18:09.202113 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.197844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33827 (local address 127.0.0.1:40414)
0620 00:18:09.198376 (+ 532us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.198382 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.198417 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.198668 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.198668 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.198670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:09.198734 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.198735 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.200061 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.200062 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.201113 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.201115 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.201791 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.201792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.201875 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.201889 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.201979 (+ 90us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.202036 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":86,"threads_started":1}
I20260620 00:18:09.202183 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.197919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55296 (local address 127.0.98.2:33767)
0620 00:18:09.198094 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.198100 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.198255 (+ 155us) server_negotiation.cc:407] Connection header received
0620 00:18:09.198371 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.198373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.198387 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.198488 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:09.198832 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.199554 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.200950 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.201691 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.201760 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.201818 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.202041 (+ 223us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.202094 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260620 00:18:09.202342 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.197844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42251 (local address 127.0.0.1:50064)
0620 00:18:09.198488 (+ 644us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.198496 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.198536 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.198812 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.198813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.198815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:09.198861 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.198862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.200111 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.200112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.201168 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.201171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.202118 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.202119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.202178 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.202182 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.202232 (+ 50us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.202272 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260620 00:18:09.202452 4954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.197818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33767 (local address 127.0.0.1:55296)
0620 00:18:09.198130 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.198181 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.198265 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.198650 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.198654 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.198657 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:09.198732 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.198734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.199657 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.199658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.200864 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.200867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.201852 (+ 985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.201853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.201925 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.201929 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.202290 (+ 361us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.202341 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":5,"thread_start_us":48,"threads_started":1}
I20260620 00:18:09.202631 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.198083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50064 (local address 127.0.98.1:42251)
0620 00:18:09.198239 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.198244 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.198592 (+ 348us) server_negotiation.cc:407] Connection header received
0620 00:18:09.198634 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.198636 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.198651 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.198723 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:09.198944 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.200013 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.201260 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.201978 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.202036 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.202086 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.202519 (+ 433us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.202561 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260620 00:18:09.203361 4610 tablet_service.cc:1467] Processing CreateTablet for tablet 594ff4d25b4e4935af6e47a261f867bb (DEFAULT_TABLE table=test-table [id=03869255625c4092961150834a7c26f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:09.203361 4743 tablet_service.cc:1467] Processing CreateTablet for tablet 594ff4d25b4e4935af6e47a261f867bb (DEFAULT_TABLE table=test-table [id=03869255625c4092961150834a7c26f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:09.203742 4743 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594ff4d25b4e4935af6e47a261f867bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:09.203742 4610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594ff4d25b4e4935af6e47a261f867bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:09.207283 4959 tablet_bootstrap.cc:492] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: Bootstrap starting.
I20260620 00:18:09.207702 4876 tablet_service.cc:1467] Processing CreateTablet for tablet 594ff4d25b4e4935af6e47a261f867bb (DEFAULT_TABLE table=test-table [id=03869255625c4092961150834a7c26f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:09.207984 4959 tablet_bootstrap.cc:654] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:09.208093 4876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 594ff4d25b4e4935af6e47a261f867bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:09.208585 4959 log.cc:824] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:09.208935 4960 tablet_bootstrap.cc:492] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5: Bootstrap starting.
I20260620 00:18:09.209641 4960 tablet_bootstrap.cc:654] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:09.210137 4960 log.cc:824] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:09.210675 4959 tablet_bootstrap.cc:492] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: No bootstrap required, opened a new log
I20260620 00:18:09.210755 4959 ts_tablet_manager.cc:1397] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 00:18:09.210961 4960 tablet_bootstrap.cc:492] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5: No bootstrap required, opened a new log
I20260620 00:18:09.211027 4960 ts_tablet_manager.cc:1397] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:18:09.211298 4963 tablet_bootstrap.cc:492] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: Bootstrap starting.
I20260620 00:18:09.211894 4963 tablet_bootstrap.cc:654] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:09.212508 4963 log.cc:824] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:09.212917 4959 raft_consensus.cc:348] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } }
I20260620 00:18:09.213124 4959 raft_consensus.cc:374] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:09.213160 4959 raft_consensus.cc:729] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d00bcb99cf46348a0c97fdaeabef19, State: Initialized, Role: FOLLOWER
I20260620 00:18:09.213080 4960 raft_consensus.cc:348] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } }
I20260620 00:18:09.213258 4960 raft_consensus.cc:374] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:09.213289 4963 tablet_bootstrap.cc:492] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: No bootstrap required, opened a new log
I20260620 00:18:09.213308 4960 raft_consensus.cc:729] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a126afe53fb74119adb91b63106d69e5, State: Initialized, Role: FOLLOWER
I20260620 00:18:09.213349 4963 ts_tablet_manager.cc:1397] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:18:09.213344 4959 consensus_queue.cc:260] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } }
I20260620 00:18:09.213439 4960 consensus_queue.cc:260] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } }
I20260620 00:18:09.213670 4959 ts_tablet_manager.cc:1428] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 00:18:09.213735 4960 ts_tablet_manager.cc:1428] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:18:09.213811 4809 heartbeater.cc:503] Master 127.0.98.62:34645 was elected leader, sending a full tablet report...
I20260620 00:18:09.214076 4676 heartbeater.cc:503] Master 127.0.98.62:34645 was elected leader, sending a full tablet report...
I20260620 00:18:09.215457 4963 raft_consensus.cc:348] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } }
I20260620 00:18:09.215584 4963 raft_consensus.cc:374] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:09.215605 4963 raft_consensus.cc:729] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4bcbefb8f44edeb9d76d3e2064266d, State: Initialized, Role: FOLLOWER
I20260620 00:18:09.215734 4963 consensus_queue.cc:260] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } }
I20260620 00:18:09.216001 4963 ts_tablet_manager.cc:1428] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:18:09.216086 4942 heartbeater.cc:503] Master 127.0.98.62:34645 was elected leader, sending a full tablet report...
W20260620 00:18:09.252769 4810 tablet.cc:2369] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:18:09.288975 4966 raft_consensus.cc:482] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:09.289106 4966 raft_consensus.cc:504] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } }
I20260620 00:18:09.289444 4966 leader_election.cc:288] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 41d00bcb99cf46348a0c97fdaeabef19 (127.0.98.2:33767), 3b4bcbefb8f44edeb9d76d3e2064266d (127.0.98.3:33827)
I20260620 00:18:09.293736 4678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.289638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33767 (local address 127.0.98.1:45513)
0620 00:18:09.289717 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.289729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.289771 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.290014 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.290016 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.290019 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:09.290101 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.290103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.291128 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.291130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.292684 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.292686 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.293429 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.293431 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.293544 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.293547 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.293603 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.293652 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55,"mutex_wait_us":68}
I20260620 00:18:09.293953 4968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.289661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33827 (local address 127.0.98.1:38731)
0620 00:18:09.289914 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.289923 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.290020 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.290203 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.290203 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.290205 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:09.290270 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.290272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.291150 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.291151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.292673 (+ 1522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:09.292677 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.293643 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.293644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:09.293771 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.293775 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.293828 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.293869 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216,"mutex_wait_us":1,"thread_start_us":125,"threads_started":1}
I20260620 00:18:09.294057 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.289764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45513 (local address 127.0.98.2:33767)
0620 00:18:09.289863 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.289865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.289872 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:18:09.289898 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.289899 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.289911 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.289968 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:09.290204 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.290969 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.292767 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.293297 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.293377 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.293392 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.293934 (+ 542us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.293994 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260620 00:18:09.294152 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.289804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38731 (local address 127.0.98.3:33827)
0620 00:18:09.289854 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.289856 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.290022 (+ 166us) server_negotiation.cc:407] Connection header received
0620 00:18:09.290111 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.290112 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.290122 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.290198 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:09.290345 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.291045 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.292820 (+ 1775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.293555 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.293630 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.293642 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.294034 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.294079 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:18:09.294544 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594ff4d25b4e4935af6e47a261f867bb" candidate_uuid: "a126afe53fb74119adb91b63106d69e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d00bcb99cf46348a0c97fdaeabef19" is_pre_election: true
I20260620 00:18:09.294716 4763 raft_consensus.cc:2454] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a126afe53fb74119adb91b63106d69e5 in term 0.
I20260620 00:18:09.295011 4565 leader_election.cc:302] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d00bcb99cf46348a0c97fdaeabef19, a126afe53fb74119adb91b63106d69e5; no voters:
I20260620 00:18:09.295143 4966 raft_consensus.cc:2781] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:09.295207 4966 raft_consensus.cc:482] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:09.295240 4966 raft_consensus.cc:3037] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:09.295432 4896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594ff4d25b4e4935af6e47a261f867bb" candidate_uuid: "a126afe53fb74119adb91b63106d69e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" is_pre_election: true
I20260620 00:18:09.295584 4896 raft_consensus.cc:2454] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a126afe53fb74119adb91b63106d69e5 in term 0.
I20260620 00:18:09.296365 4966 raft_consensus.cc:504] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } }
I20260620 00:18:09.296526 4966 leader_election.cc:288] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [CANDIDATE]: Term 1 election: Requested vote from peers 41d00bcb99cf46348a0c97fdaeabef19 (127.0.98.2:33767), 3b4bcbefb8f44edeb9d76d3e2064266d (127.0.98.3:33827)
I20260620 00:18:09.296667 4763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594ff4d25b4e4935af6e47a261f867bb" candidate_uuid: "a126afe53fb74119adb91b63106d69e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "41d00bcb99cf46348a0c97fdaeabef19"
I20260620 00:18:09.296738 4763 raft_consensus.cc:3037] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:09.296710 4896 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "594ff4d25b4e4935af6e47a261f867bb" candidate_uuid: "a126afe53fb74119adb91b63106d69e5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d"
I20260620 00:18:09.296816 4896 raft_consensus.cc:3037] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:09.297768 4763 raft_consensus.cc:2454] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a126afe53fb74119adb91b63106d69e5 in term 1.
I20260620 00:18:09.297955 4565 leader_election.cc:302] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 41d00bcb99cf46348a0c97fdaeabef19, a126afe53fb74119adb91b63106d69e5; no voters:
I20260620 00:18:09.297957 4896 raft_consensus.cc:2454] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a126afe53fb74119adb91b63106d69e5 in term 1.
I20260620 00:18:09.298050 4966 raft_consensus.cc:2781] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:09.298241 4966 raft_consensus.cc:686] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [term 1 LEADER]: Becoming Leader. State: Replica: a126afe53fb74119adb91b63106d69e5, State: Running, Role: LEADER
I20260620 00:18:09.298345 4966 consensus_queue.cc:237] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } }
I20260620 00:18:09.299355 4485 catalog_manager.cc:5495] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 reported cstate change: term changed from 0 to 1, leader changed from <none> to a126afe53fb74119adb91b63106d69e5 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a126afe53fb74119adb91b63106d69e5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a126afe53fb74119adb91b63106d69e5" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42251 } health_report { overall_health: HEALTHY } } }
I20260620 00:18:09.317960 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.315533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50080 (local address 127.0.98.1:42251)
0620 00:18:09.315613 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.315615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.315624 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:09.315660 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.315661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.315683 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.315754 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:09.315904 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.316456 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.317215 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:09.317400 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:09.317490 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:09.317651 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:09.317763 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:09.317829 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.317864 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.317882 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 00:18:09.320063 4763 raft_consensus.cc:1261] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [term 1 FOLLOWER]: Refusing update from remote peer a126afe53fb74119adb91b63106d69e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:09.320083 4896 raft_consensus.cc:1261] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [term 1 FOLLOWER]: Refusing update from remote peer a126afe53fb74119adb91b63106d69e5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:09.320479 4966 consensus_queue.cc:1035] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b4bcbefb8f44edeb9d76d3e2064266d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33827 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:09.320689 4969 consensus_queue.cc:1035] T 594ff4d25b4e4935af6e47a261f867bb P a126afe53fb74119adb91b63106d69e5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "41d00bcb99cf46348a0c97fdaeabef19" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:09.323410 4973 mvcc.cc:205] Tried to move back new op lower bound from 7298722567451435008 to 7298722567366832128. Current Snapshot: MvccSnapshot[applied={T|T < 7298722567451435008}]
I20260620 00:18:09.324322 4974 mvcc.cc:205] Tried to move back new op lower bound from 7298722567451435008 to 7298722567366832128. Current Snapshot: MvccSnapshot[applied={T|T < 7298722567451435008 or (T in {7298722567451435008})}]
I20260620 00:18:09.327023 4486 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:09.327142 4486 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:09.327931 4486 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 67eb9463ee8e4e14b30983f8f4891a73: Sending DeleteTablet for 3 replicas of tablet 594ff4d25b4e4935af6e47a261f867bb
I20260620 00:18:09.328315 4743 tablet_service.cc:1514] Processing DeleteTablet for tablet 594ff4d25b4e4935af6e47a261f867bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55296
I20260620 00:18:09.328332 4876 tablet_service.cc:1514] Processing DeleteTablet for tablet 594ff4d25b4e4935af6e47a261f867bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:40414
I20260620 00:18:09.328583 4610 tablet_service.cc:1514] Processing DeleteTablet for tablet 594ff4d25b4e4935af6e47a261f867bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:09 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50064
I20260620 00:18:09.328678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 4548
I20260620 00:18:09.328665 4981 tablet_replica.cc:330] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: stopping tablet replica
I20260620 00:18:09.329329 4981 raft_consensus.cc:2229] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:09.329543 4981 raft_consensus.cc:2258] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:09.330013 4981 ts_tablet_manager.cc:1905] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:09.330072 4982 tablet_replica.cc:330] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: stopping tablet replica
I20260620 00:18:09.330173 4982 raft_consensus.cc:2229] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:09.330278 4982 raft_consensus.cc:2258] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:09.331645 4981 ts_tablet_manager.cc:1918] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:09.331692 4982 ts_tablet_manager.cc:1905] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:09.331744 4981 log.cc:1196] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/594ff4d25b4e4935af6e47a261f867bb
I20260620 00:18:09.332029 4981 ts_tablet_manager.cc:1939] T 594ff4d25b4e4935af6e47a261f867bb P 41d00bcb99cf46348a0c97fdaeabef19: Deleting consensus metadata
I20260620 00:18:09.332525 4472 catalog_manager.cc:4841] TS 41d00bcb99cf46348a0c97fdaeabef19 (127.0.98.2:33767): tablet 594ff4d25b4e4935af6e47a261f867bb (table test-table [id=03869255625c4092961150834a7c26f0]) successfully deleted
I20260620 00:18:09.333215 4982 ts_tablet_manager.cc:1918] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:09.333305 4982 log.cc:1196] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/594ff4d25b4e4935af6e47a261f867bb
I20260620 00:18:09.333556 4982 ts_tablet_manager.cc:1939] T 594ff4d25b4e4935af6e47a261f867bb P 3b4bcbefb8f44edeb9d76d3e2064266d: Deleting consensus metadata
I20260620 00:18:09.333969 4471 catalog_manager.cc:4841] TS 3b4bcbefb8f44edeb9d76d3e2064266d (127.0.98.3:33827): tablet 594ff4d25b4e4935af6e47a261f867bb (table test-table [id=03869255625c4092961150834a7c26f0]) successfully deleted
W20260620 00:18:09.334571 4470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42251 (error 108)
I20260620 00:18:09.334846 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 4681
I20260620 00:18:09.335474 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.334960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42251 (local address 127.0.0.1:50086)
0620 00:18:09.335405 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.335446 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42251: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424}
W20260620 00:18:09.335599 4470 catalog_manager.cc:4584] TS a126afe53fb74119adb91b63106d69e5 (127.0.98.1:42251): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 594ff4d25b4e4935af6e47a261f867bb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42251: connect: Connection refused (error 111)
I20260620 00:18:09.340202 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 4814
I20260620 00:18:09.345371 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 4455
2026-06-20T00:18:09Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2289 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-06-20T00:18:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:09Z Disabled control of system clock
I20260620 00:18:09.374150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40883
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:18:09.456833 4988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:09.457074 4988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:09.457098 4988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:09.458309 4988 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:09.458350 4988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:09.458364 4988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:09.458376 4988 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:09.458388 4988 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:09.459710 4988 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40883
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:09.459911 4988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:09.460178 4988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:09.464932 4994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.464921 4996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.465337 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:09.465639 4988 server_base.cc:1034] running on GCE node
I20260620 00:18:09.465922 4988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:09.466238 4988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:09.467410 4988 hybrid_clock.cc:648] HybridClock initialized: now 1781914689467379 us; error 59 us; skew 500 ppm
I20260620 00:18:09.468762 4988 webserver.cc:460] Webserver started at http://127.0.98.62:40595/ using document root <none> and password file <none>
I20260620 00:18:09.469022 4988 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:09.469091 4988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:09.469265 4988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:09.470323 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "68b855a5120c4d3d863709c5ee0edf7a"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "b9b3a235a72911cb9f0468039a181f78"
server_key_iv: "ac4378dddba359defae9f3187d46f2d5"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.470682 4988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "68b855a5120c4d3d863709c5ee0edf7a"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "b9b3a235a72911cb9f0468039a181f78"
server_key_iv: "ac4378dddba359defae9f3187d46f2d5"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.471975 4988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:18:09.472800 5002 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.472967 4988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:09.473066 4988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "68b855a5120c4d3d863709c5ee0edf7a"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "b9b3a235a72911cb9f0468039a181f78"
server_key_iv: "ac4378dddba359defae9f3187d46f2d5"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.473147 4988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:09.476523 4988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:09.476790 4988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:09.476929 4988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:09.480677 4988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40883
I20260620 00:18:09.480744 5054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40883 every 8 connection(s)
I20260620 00:18:09.481076 4988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:09.481779 5055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:09.484712 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a: Bootstrap starting.
I20260620 00:18:09.485309 5055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:09.485618 5055 log.cc:824] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:09.486371 5055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a: No bootstrap required, opened a new log
I20260620 00:18:09.488116 5055 raft_consensus.cc:348] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b855a5120c4d3d863709c5ee0edf7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260620 00:18:09.488361 5055 raft_consensus.cc:374] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:09.488391 5055 raft_consensus.cc:729] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68b855a5120c4d3d863709c5ee0edf7a, State: Initialized, Role: FOLLOWER
I20260620 00:18:09.488583 5055 consensus_queue.cc:260] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b855a5120c4d3d863709c5ee0edf7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260620 00:18:09.488656 5055 raft_consensus.cc:388] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:09.488715 5055 raft_consensus.cc:482] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:09.488785 5055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:09.488839 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 4988
I20260620 00:18:09.488941 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:09.489131 392 external_mini_cluster.cc:1428] Setting key 9399881f8d033be1b52e4229b0323552
I20260620 00:18:09.489717 5055 raft_consensus.cc:504] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b855a5120c4d3d863709c5ee0edf7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260620 00:18:09.489849 5055 leader_election.cc:302] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 68b855a5120c4d3d863709c5ee0edf7a; no voters:
I20260620 00:18:09.490098 5055 leader_election.cc:288] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:09.490134 5060 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:09.490259 5060 raft_consensus.cc:686] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [term 1 LEADER]: Becoming Leader. State: Replica: 68b855a5120c4d3d863709c5ee0edf7a, State: Running, Role: LEADER
I20260620 00:18:09.490352 5060 consensus_queue.cc:237] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b855a5120c4d3d863709c5ee0edf7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } }
I20260620 00:18:09.490437 5055 sys_catalog.cc:564] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:09.490986 5062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 68b855a5120c4d3d863709c5ee0edf7a. Latest consensus state: current_term: 1 leader_uuid: "68b855a5120c4d3d863709c5ee0edf7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b855a5120c4d3d863709c5ee0edf7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } }
I20260620 00:18:09.491279 5062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:09.491199 5061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "68b855a5120c4d3d863709c5ee0edf7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "68b855a5120c4d3d863709c5ee0edf7a" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40883 } } }
I20260620 00:18:09.491571 5061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:09.492115 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.489814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44518 (local address 127.0.98.62:40883)
0620 00:18:09.490078 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.490086 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.490111 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:18:09.490287 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.490296 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.490352 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.490814 (+ 462us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.491420 (+ 606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.491424 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.491425 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:09.491429 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.491472 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:09.491486 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.491768 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.491803 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.491863 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":97,"threads_started":1}
W20260620 00:18:09.492523 5076 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:18:09.492576 5076 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:18:09.492621 5069 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:09.492794 5069 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:09.494328 5069 catalog_manager.cc:1347] Generated new cluster ID: 761cab27673641bc9a5d601d83114080
I20260620 00:18:09.494383 5069 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:09.504915 5069 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:09.505110 5069 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:09.509477 5069 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a: Generated new TSK 0
I20260620 00:18:09.509691 5069 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:18:09.513554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--builtin_ntp_servers=127.0.98.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:18:09.602237 5079 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:09.602492 5079 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:09.602530 5079 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:09.603790 5079 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:09.603863 5079 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:09.603911 5079 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:09.605425 5079 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5079
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:09.605715 5079 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:09.606024 5079 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:09.612038 5084 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.612082 5085 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.612208 5087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:09.612334 5079 server_base.cc:1034] running on GCE node
I20260620 00:18:09.612735 5079 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:09.613015 5079 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:09.614209 5079 hybrid_clock.cc:648] HybridClock initialized: now 1781914689614183 us; error 47 us; skew 500 ppm
I20260620 00:18:09.615572 5079 webserver.cc:460] Webserver started at http://127.0.98.1:46163/ using document root <none> and password file <none>
I20260620 00:18:09.615850 5079 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:09.615939 5079 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:09.616104 5079 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:09.617331 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "a9d9961e397f4f64ade5813362eb833f"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "2d1f730aac27faf1a8366271e5a83426"
server_key_iv: "9d4477fe6f7acf59dab8eaf6008733cd"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.617718 5079 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a9d9961e397f4f64ade5813362eb833f"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "2d1f730aac27faf1a8366271e5a83426"
server_key_iv: "9d4477fe6f7acf59dab8eaf6008733cd"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.619201 5079 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:09.619982 5093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.620191 5079 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:09.620285 5079 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "a9d9961e397f4f64ade5813362eb833f"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "2d1f730aac27faf1a8366271e5a83426"
server_key_iv: "9d4477fe6f7acf59dab8eaf6008733cd"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.620364 5079 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:09.623708 5079 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:09.624029 5079 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:09.624207 5079 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:09.624444 5079 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:09.624774 5079 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:09.624830 5079 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.624869 5079 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:09.624897 5079 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.630419 5079 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33933
I20260620 00:18:09.630484 5206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33933 every 8 connection(s)
I20260620 00:18:09.630854 5079 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:18:09.634418 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.631684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43697 (local address 127.0.98.62:40883)
0620 00:18:09.631757 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.631759 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.632092 (+ 333us) server_negotiation.cc:407] Connection header received
0620 00:18:09.632756 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.632757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.632763 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.632814 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.634163 (+ 1349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.634165 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.634165 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:09.634169 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.634185 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:09.634190 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.634250 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.634349 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.634363 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:18:09.634622 5209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.631726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.1:43697)
0620 00:18:09.631948 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.631993 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.632650 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.632894 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.632898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.632908 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:09.633974 (+ 1066us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:09.633975 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:09.634015 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:09.634050 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:09.634263 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:09.634268 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:09.634271 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.634351 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.634416 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":98,"threads_started":1}
I20260620 00:18:09.635110 5207 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260620 00:18:09.635218 5207 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:09.635370 5207 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260620 00:18:09.635928 5019 ts_manager.cc:194] Registered new tserver with Master: a9d9961e397f4f64ade5813362eb833f (127.0.98.1:33933)
I20260620 00:18:09.639078 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 5079
I20260620 00:18:09.639206 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:09.639425 392 external_mini_cluster.cc:1428] Setting key 07355920860dd0db821c485bcf821e0c
I20260620 00:18:09.640378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--builtin_ntp_servers=127.0.98.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:18:09.728252 5210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:09.728520 5210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:09.728562 5210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:09.730033 5210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:09.730124 5210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:09.730170 5210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:09.731678 5210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:09.731978 5210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:09.732334 5210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:09.737528 5216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.737535 5218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.737548 5215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:09.737867 5210 server_base.cc:1034] running on GCE node
I20260620 00:18:09.738094 5210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:09.738377 5210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:09.739600 5210 hybrid_clock.cc:648] HybridClock initialized: now 1781914689739583 us; error 39 us; skew 500 ppm
I20260620 00:18:09.741092 5210 webserver.cc:460] Webserver started at http://127.0.98.2:37309/ using document root <none> and password file <none>
I20260620 00:18:09.741357 5210 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:09.741420 5210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:09.741559 5210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:09.742682 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "d7d1795dae417ffc2c3ee482a33f48c2"
server_key_iv: "06c616ebdb234132cad93a9614f04b14"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.743054 5210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "d7d1795dae417ffc2c3ee482a33f48c2"
server_key_iv: "06c616ebdb234132cad93a9614f04b14"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.744509 5210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:09.745450 5224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.745699 5210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:09.745800 5210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "d7d1795dae417ffc2c3ee482a33f48c2"
server_key_iv: "06c616ebdb234132cad93a9614f04b14"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.745862 5210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:09.749440 5210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:09.749712 5210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:09.749816 5210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:09.749997 5210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:09.750326 5210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:09.750382 5210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.750445 5210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:09.750475 5210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.756376 5210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35239
I20260620 00:18:09.756418 5337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35239 every 8 connection(s)
I20260620 00:18:09.756772 5210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:09.760214 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.757617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48737 (local address 127.0.98.62:40883)
0620 00:18:09.757672 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.757675 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.757907 (+ 232us) server_negotiation.cc:407] Connection header received
0620 00:18:09.758405 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.758406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.758412 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.758472 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.759872 (+ 1400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.759873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.759873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:09.759878 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.759895 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:09.759900 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.759959 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.760073 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.760088 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:18:09.760416 5340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.757518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.2:48737)
0620 00:18:09.757759 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.757797 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.758290 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.758569 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.758573 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.758582 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:09.759695 (+ 1113us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:09.759695 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:09.759728 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:09.759762 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:09.759967 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:09.759970 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:09.759973 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.760037 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.760093 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":71,"threads_started":1}
I20260620 00:18:09.760968 5338 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260620 00:18:09.761087 5338 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:09.761271 5338 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260620 00:18:09.761696 5019 ts_manager.cc:194] Registered new tserver with Master: e2a4edf4ba7c42ac9dd4a02ab568bea6 (127.0.98.2:35239)
I20260620 00:18:09.765655 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 5210
I20260620 00:18:09.765774 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:09.766036 392 external_mini_cluster.cc:1428] Setting key fdfb5377846b55d60614cea8891562e8
I20260620 00:18:09.767026 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--builtin_ntp_servers=127.0.98.20:34329
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260620 00:18:09.849197 5341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:09.849423 5341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:09.849449 5341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:09.850768 5341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:09.850817 5341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:09.850847 5341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:09.852377 5341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34329
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40883
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5341
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:09.852646 5341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:09.852957 5341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:09.858227 5349 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.858326 5346 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:09.858338 5347 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:09.858474 5341 server_base.cc:1034] running on GCE node
I20260620 00:18:09.858737 5341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:09.859035 5341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:09.860242 5341 hybrid_clock.cc:648] HybridClock initialized: now 1781914689860236 us; error 45 us; skew 500 ppm
I20260620 00:18:09.861621 5341 webserver.cc:460] Webserver started at http://127.0.98.3:46337/ using document root <none> and password file <none>
I20260620 00:18:09.861883 5341 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:09.861958 5341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:09.862111 5341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:09.863226 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "e5dc837934924b00aeb836650472bb8e"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "761dd224392a925f87bb1aba4dc54586"
server_key_iv: "13eabc6d76c93be1a5177998ef2d0925"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.863583 5341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e5dc837934924b00aeb836650472bb8e"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "761dd224392a925f87bb1aba4dc54586"
server_key_iv: "13eabc6d76c93be1a5177998ef2d0925"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.865044 5341 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:09.865852 5355 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.866067 5341 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:09.866163 5341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "e5dc837934924b00aeb836650472bb8e"
format_stamp: "Formatted at 2026-06-20 00:18:09 on dist-test-slave-r81b"
server_key: "761dd224392a925f87bb1aba4dc54586"
server_key_iv: "13eabc6d76c93be1a5177998ef2d0925"
server_key_version: "encryptionkey@0"
I20260620 00:18:09.866219 5341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:09.869580 5341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:09.869804 5341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:09.869900 5341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:09.870074 5341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:09.870311 5341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:09.870342 5341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.870362 5341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:09.870378 5341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:09.876063 5341 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44247
I20260620 00:18:09.876163 5468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44247 every 8 connection(s)
I20260620 00:18:09.876498 5341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:09.879480 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.877251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43207 (local address 127.0.98.62:40883)
0620 00:18:09.877338 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.877340 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.877583 (+ 243us) server_negotiation.cc:407] Connection header received
0620 00:18:09.878052 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.878053 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.878058 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.878110 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.879308 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.879309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.879309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:09.879312 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.879322 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:09.879326 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.879369 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.879414 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.879426 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 00:18:09.879712 5471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.877258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40883 (local address 127.0.98.3:43207)
0620 00:18:09.877454 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.877489 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.877968 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.878164 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.878167 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.878176 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:09.879172 (+ 996us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:09.879172 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:09.879202 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:09.879233 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:09.879367 (+ 134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:09.879369 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:09.879371 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.879423 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.879472 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
I20260620 00:18:09.880236 5469 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40883
I20260620 00:18:09.880344 5469 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:09.880502 5469 heartbeater.cc:511] Master 127.0.98.62:40883 requested a full tablet report, sending...
I20260620 00:18:09.880919 5019 ts_manager.cc:194] Registered new tserver with Master: e5dc837934924b00aeb836650472bb8e (127.0.98.3:44247)
I20260620 00:18:09.881919 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 5341
I20260620 00:18:09.882009 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:09.882153 392 external_mini_cluster.cc:1428] Setting key 5c37f80e1300b875ad91309067ef6fac
I20260620 00:18:09.882987 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:18:09.885603 5059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.884376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44534 (local address 127.0.98.62:40883)
0620 00:18:09.884438 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.884440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.884582 (+ 142us) server_negotiation.cc:407] Connection header received
0620 00:18:09.884620 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.884621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.884626 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.884682 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.885308 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.885309 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.885309 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:09.885313 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.885329 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:09.885333 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.885390 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.885530 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.885546 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:18:09.887602 5019 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44534:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:18:09.889364 5019 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:09.893865 5477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.891982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44247 (local address 127.0.0.1:43468)
0620 00:18:09.892286 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.892331 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.892396 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.892779 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.892785 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.892793 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:09.893396 (+ 603us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:09.893397 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:09.893417 (+ 20us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:09.893446 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:09.893681 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:09.893687 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:09.893690 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.893752 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.893766 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":218,"threads_started":1}
I20260620 00:18:09.893873 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.892346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43468 (local address 127.0.98.3:44247)
0620 00:18:09.892581 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.892587 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.892599 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:18:09.892628 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.892632 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.892644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.892752 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.893536 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.893546 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.893546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:09.893551 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.893591 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:09.893626 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.893689 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.893769 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.893786 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
I20260620 00:18:09.894615 5479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.892454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33933 (local address 127.0.0.1:56878)
0620 00:18:09.893563 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.893573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.893619 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.893799 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.893799 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.893801 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:09.894206 (+ 405us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:09.894207 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:09.894214 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:09.894228 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:09.894465 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:09.894466 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:09.894466 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.894532 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.894546 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"thread_start_us":89,"threads_started":1}
I20260620 00:18:09.894639 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.892657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56878 (local address 127.0.98.1:33933)
0620 00:18:09.892873 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.892877 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.893682 (+ 805us) server_negotiation.cc:407] Connection header received
0620 00:18:09.893712 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.893714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.893723 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.893796 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.894321 (+ 525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.894329 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.894329 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:09.894334 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.894367 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:09.894396 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.894470 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.894538 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.894555 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":108,"threads_started":1}
I20260620 00:18:09.894695 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.892070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.0.98.2:35239)
0620 00:18:09.893085 (+ 1015us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:09.893088 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:09.893096 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:18:09.893151 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:09.893154 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:09.893162 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:09.893239 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:09.894334 (+ 1095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:09.894342 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:09.894342 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:09.894346 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:09.894379 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:09.894405 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:09.894469 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:09.894572 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:18:09.894587 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":981,"thread_start_us":168,"threads_started":1}
I20260620 00:18:09.894869 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:09.891982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35239 (local address 127.0.0.1:51838)
0620 00:18:09.892608 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:09.892626 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:09.892670 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:09.893238 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:09.893239 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:09.893241 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:09.894244 (+ 1003us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:09.894245 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:09.894250 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:09.894260 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:09.894494 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:09.894495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:09.894495 (+ 0us) client_negotiation.cc:769] Sending connection context
0620 00:18:09.894797 (+ 302us) client_negotiation.cc:240] Negotiation successful
0620 00:18:09.894810 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":609,"mutex_wait_us":22,"thread_start_us":229,"threads_started":1}
I20260620 00:18:09.894905 5403 tablet_service.cc:1467] Processing CreateTablet for tablet 00adc24ec48a48fa829fcfcb89beaea4 (DEFAULT_TABLE table=test-table [id=8a4db7064157481abb4a5b5cf5c382e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:09.895306 5403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00adc24ec48a48fa829fcfcb89beaea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:09.895270 5141 tablet_service.cc:1467] Processing CreateTablet for tablet 00adc24ec48a48fa829fcfcb89beaea4 (DEFAULT_TABLE table=test-table [id=8a4db7064157481abb4a5b5cf5c382e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:09.895539 5141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00adc24ec48a48fa829fcfcb89beaea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:09.895625 5272 tablet_service.cc:1467] Processing CreateTablet for tablet 00adc24ec48a48fa829fcfcb89beaea4 (DEFAULT_TABLE table=test-table [id=8a4db7064157481abb4a5b5cf5c382e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:09.896055 5272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00adc24ec48a48fa829fcfcb89beaea4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:09.898411 5483 tablet_bootstrap.cc:492] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f: Bootstrap starting.
I20260620 00:18:09.899122 5483 tablet_bootstrap.cc:654] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:09.899217 5484 tablet_bootstrap.cc:492] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: Bootstrap starting.
I20260620 00:18:09.899456 5483 log.cc:824] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:09.899758 5485 tablet_bootstrap.cc:492] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: Bootstrap starting.
I20260620 00:18:09.899889 5484 tablet_bootstrap.cc:654] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:09.900290 5484 log.cc:824] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:09.900425 5485 tablet_bootstrap.cc:654] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:09.900768 5485 log.cc:824] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:09.900882 5483 tablet_bootstrap.cc:492] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f: No bootstrap required, opened a new log
I20260620 00:18:09.900931 5483 ts_tablet_manager.cc:1397] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:18:09.901083 5484 tablet_bootstrap.cc:492] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: No bootstrap required, opened a new log
I20260620 00:18:09.901131 5484 ts_tablet_manager.cc:1397] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:18:09.901648 5485 tablet_bootstrap.cc:492] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: No bootstrap required, opened a new log
I20260620 00:18:09.901705 5485 ts_tablet_manager.cc:1397] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:18:09.902925 5483 raft_consensus.cc:348] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } }
I20260620 00:18:09.903059 5483 raft_consensus.cc:374] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:09.903080 5483 raft_consensus.cc:729] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9d9961e397f4f64ade5813362eb833f, State: Initialized, Role: FOLLOWER
I20260620 00:18:09.903218 5483 consensus_queue.cc:260] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } }
I20260620 00:18:09.903497 5483 ts_tablet_manager.cc:1428] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:18:09.903509 5207 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
I20260620 00:18:09.903839 5484 raft_consensus.cc:348] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } }
I20260620 00:18:09.904049 5484 raft_consensus.cc:374] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:09.904088 5484 raft_consensus.cc:729] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5dc837934924b00aeb836650472bb8e, State: Initialized, Role: FOLLOWER
I20260620 00:18:09.904237 5484 consensus_queue.cc:260] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } }
I20260620 00:18:09.904510 5469 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
I20260620 00:18:09.904485 5485 raft_consensus.cc:348] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } }
I20260620 00:18:09.904646 5485 raft_consensus.cc:374] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:09.904680 5485 raft_consensus.cc:729] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2a4edf4ba7c42ac9dd4a02ab568bea6, State: Initialized, Role: FOLLOWER
I20260620 00:18:09.904812 5485 consensus_queue.cc:260] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } }
I20260620 00:18:09.905063 5485 ts_tablet_manager.cc:1428] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 00:18:09.904500 5484 ts_tablet_manager.cc:1428] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:18:09.905175 5338 heartbeater.cc:503] Master 127.0.98.62:40883 was elected leader, sending a full tablet report...
W20260620 00:18:10.007370 5339 tablet.cc:2369] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:18:10.127084 5470 tablet.cc:2369] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:18:10.131656 5208 tablet.cc:2369] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:18:10.136296 5489 raft_consensus.cc:482] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:10.136431 5489 raft_consensus.cc:504] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } }
I20260620 00:18:10.136924 5489 leader_election.cc:288] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e5dc837934924b00aeb836650472bb8e (127.0.98.3:44247), e2a4edf4ba7c42ac9dd4a02ab568bea6 (127.0.98.2:35239)
I20260620 00:18:10.138528 5480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.137207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50745 (local address 127.0.98.3:44247)
0620 00:18:10.137258 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:10.137260 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:10.137384 (+ 124us) server_negotiation.cc:407] Connection header received
0620 00:18:10.137442 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:10.137443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:10.137449 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:10.137492 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:10.138222 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:10.138223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:10.138223 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:10.138228 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:10.138246 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:10.138251 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:10.138318 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:10.138439 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 00:18:10.138454 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:18:10.138528 5478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.137265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50523 (local address 127.0.98.2:35239)
0620 00:18:10.137309 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:10.137311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:10.137379 (+ 68us) server_negotiation.cc:407] Connection header received
0620 00:18:10.137442 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:10.137443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:10.137449 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:10.137514 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:10.138222 (+ 708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:10.138223 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:10.138224 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:10.138228 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:10.138246 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:10.138251 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:10.138318 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:10.138439 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 00:18:10.138454 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:18:10.138593 5492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.137122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44247 (local address 127.0.98.1:50745)
0620 00:18:10.137297 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:10.137310 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:10.137377 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:10.137538 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:10.137539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:10.137558 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:10.138090 (+ 532us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:10.138090 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:10.138099 (+ 9us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:10.138125 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:10.138358 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:10.138360 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:10.138361 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:18:10.138436 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:18:10.138476 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":22,"thread_start_us":84,"threads_started":1}
I20260620 00:18:10.138803 5493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.137179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35239 (local address 127.0.98.1:50523)
0620 00:18:10.137313 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:10.137322 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:10.137377 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:10.137538 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:10.137539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:10.137558 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:10.138126 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:10.138127 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:10.138131 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:10.138140 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:10.138358 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:10.138360 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:10.138361 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:18:10.138438 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:18:10.138454 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":53,"threads_started":1}
I20260620 00:18:10.139024 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00adc24ec48a48fa829fcfcb89beaea4" candidate_uuid: "a9d9961e397f4f64ade5813362eb833f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5dc837934924b00aeb836650472bb8e" is_pre_election: true
I20260620 00:18:10.139117 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00adc24ec48a48fa829fcfcb89beaea4" candidate_uuid: "a9d9961e397f4f64ade5813362eb833f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" is_pre_election: true
I20260620 00:18:10.139230 5423 raft_consensus.cc:2454] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9d9961e397f4f64ade5813362eb833f in term 0.
I20260620 00:18:10.139288 5292 raft_consensus.cc:2454] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9d9961e397f4f64ade5813362eb833f in term 0.
I20260620 00:18:10.139561 5095 leader_election.cc:302] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9d9961e397f4f64ade5813362eb833f, e2a4edf4ba7c42ac9dd4a02ab568bea6; no voters:
I20260620 00:18:10.139712 5489 raft_consensus.cc:2781] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:10.139770 5489 raft_consensus.cc:482] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:10.139803 5489 raft_consensus.cc:3037] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:10.140825 5489 raft_consensus.cc:504] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } }
I20260620 00:18:10.141021 5489 leader_election.cc:288] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [CANDIDATE]: Term 1 election: Requested vote from peers e5dc837934924b00aeb836650472bb8e (127.0.98.3:44247), e2a4edf4ba7c42ac9dd4a02ab568bea6 (127.0.98.2:35239)
I20260620 00:18:10.141206 5292 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00adc24ec48a48fa829fcfcb89beaea4" candidate_uuid: "a9d9961e397f4f64ade5813362eb833f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6"
I20260620 00:18:10.141194 5423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00adc24ec48a48fa829fcfcb89beaea4" candidate_uuid: "a9d9961e397f4f64ade5813362eb833f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e5dc837934924b00aeb836650472bb8e"
I20260620 00:18:10.141297 5292 raft_consensus.cc:3037] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:10.141343 5423 raft_consensus.cc:3037] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:10.142051 5292 raft_consensus.cc:2454] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9d9961e397f4f64ade5813362eb833f in term 1.
I20260620 00:18:10.142264 5095 leader_election.cc:302] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9d9961e397f4f64ade5813362eb833f, e2a4edf4ba7c42ac9dd4a02ab568bea6; no voters:
I20260620 00:18:10.142373 5489 raft_consensus.cc:2781] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:10.142412 5423 raft_consensus.cc:2454] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9d9961e397f4f64ade5813362eb833f in term 1.
I20260620 00:18:10.142539 5489 raft_consensus.cc:686] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [term 1 LEADER]: Becoming Leader. State: Replica: a9d9961e397f4f64ade5813362eb833f, State: Running, Role: LEADER
I20260620 00:18:10.142645 5489 consensus_queue.cc:237] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } }
I20260620 00:18:10.143551 5018 catalog_manager.cc:5495] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f reported cstate change: term changed from 0 to 1, leader changed from <none> to a9d9961e397f4f64ade5813362eb833f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "a9d9961e397f4f64ade5813362eb833f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9d9961e397f4f64ade5813362eb833f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 33933 } health_report { overall_health: HEALTHY } } }
I20260620 00:18:10.179653 5482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.178612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56892 (local address 127.0.98.1:33933)
0620 00:18:10.178673 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:10.178674 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:10.178684 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:10.178775 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:10.178775 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:10.178788 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:10.178829 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:10.179398 (+ 569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:10.179399 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:10.179400 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:10.179403 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:10.179419 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:10.179423 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:10.179469 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:10.179523 (+ 54us) server_negotiation.cc:299] Negotiation successful
0620 00:18:10.179535 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:18:10.181488 5292 raft_consensus.cc:1261] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [term 1 FOLLOWER]: Refusing update from remote peer a9d9961e397f4f64ade5813362eb833f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:10.181488 5423 raft_consensus.cc:1261] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [term 1 FOLLOWER]: Refusing update from remote peer a9d9961e397f4f64ade5813362eb833f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:10.181856 5494 consensus_queue.cc:1035] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e5dc837934924b00aeb836650472bb8e" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44247 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:10.182004 5489 consensus_queue.cc:1035] T 00adc24ec48a48fa829fcfcb89beaea4 P a9d9961e397f4f64ade5813362eb833f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2a4edf4ba7c42ac9dd4a02ab568bea6" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:10.185382 5499 mvcc.cc:205] Tried to move back new op lower bound from 7298722570980855808 to 7298722570825068544. Current Snapshot: MvccSnapshot[applied={T|T < 7298722570980855808 or (T in {7298722570980855808})}]
I20260620 00:18:10.185570 5498 mvcc.cc:205] Tried to move back new op lower bound from 7298722570980855808 to 7298722570825068544. Current Snapshot: MvccSnapshot[applied={T|T < 7298722570980855808}]
I20260620 00:18:10.187196 5019 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:10.187296 5019 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44534:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:10.188182 5019 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 68b855a5120c4d3d863709c5ee0edf7a: Sending DeleteTablet for 3 replicas of tablet 00adc24ec48a48fa829fcfcb89beaea4
I20260620 00:18:10.188585 5272 tablet_service.cc:1514] Processing DeleteTablet for tablet 00adc24ec48a48fa829fcfcb89beaea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:10 UTC) from {username='slave'} at 127.0.0.1:51838
I20260620 00:18:10.188683 5403 tablet_service.cc:1514] Processing DeleteTablet for tablet 00adc24ec48a48fa829fcfcb89beaea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:10 UTC) from {username='slave'} at 127.0.0.1:43468
I20260620 00:18:10.188753 5141 tablet_service.cc:1514] Processing DeleteTablet for tablet 00adc24ec48a48fa829fcfcb89beaea4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:10 UTC) from {username='slave'} at 127.0.0.1:56878
I20260620 00:18:10.188884 5507 tablet_replica.cc:330] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: stopping tablet replica
I20260620 00:18:10.188979 5508 tablet_replica.cc:330] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: stopping tablet replica
I20260620 00:18:10.189091 5507 raft_consensus.cc:2229] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:10.189272 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 5079
I20260620 00:18:10.189355 5507 raft_consensus.cc:2258] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:10.190375 5507 ts_tablet_manager.cc:1905] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:10.192525 5507 ts_tablet_manager.cc:1918] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:10.192605 5507 log.cc:1196] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/00adc24ec48a48fa829fcfcb89beaea4
I20260620 00:18:10.192866 5507 ts_tablet_manager.cc:1939] T 00adc24ec48a48fa829fcfcb89beaea4 P e2a4edf4ba7c42ac9dd4a02ab568bea6: Deleting consensus metadata
I20260620 00:18:10.193334 5004 catalog_manager.cc:4841] TS e2a4edf4ba7c42ac9dd4a02ab568bea6 (127.0.98.2:35239): tablet 00adc24ec48a48fa829fcfcb89beaea4 (table test-table [id=8a4db7064157481abb4a5b5cf5c382e8]) successfully deleted
I20260620 00:18:10.189078 5508 raft_consensus.cc:2229] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260620 00:18:10.196028 5005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:33933 (error 108)
I20260620 00:18:10.196322 5508 raft_consensus.cc:2258] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:10.196583 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.196456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:33933 (local address 127.0.0.1:56900)
0620 00:18:10.196501 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:10.196544 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260620 00:18:10.196687 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 5210
I20260620 00:18:10.196938 5508 ts_tablet_manager.cc:1905] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 00:18:10.197999 5005 catalog_manager.cc:4584] TS a9d9961e397f4f64ade5813362eb833f (127.0.98.1:33933): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00adc24ec48a48fa829fcfcb89beaea4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:33933: connect: Connection refused (error 111)
I20260620 00:18:10.199142 5508 ts_tablet_manager.cc:1918] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:10.199235 5508 log.cc:1196] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/00adc24ec48a48fa829fcfcb89beaea4
I20260620 00:18:10.199527 5508 ts_tablet_manager.cc:1939] T 00adc24ec48a48fa829fcfcb89beaea4 P e5dc837934924b00aeb836650472bb8e: Deleting consensus metadata
I20260620 00:18:10.200161 5005 catalog_manager.cc:4841] TS e5dc837934924b00aeb836650472bb8e (127.0.98.3:44247): tablet 00adc24ec48a48fa829fcfcb89beaea4 (table test-table [id=8a4db7064157481abb4a5b5cf5c382e8]) successfully deleted
I20260620 00:18:10.202486 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 5341
I20260620 00:18:10.207891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 4988
2026-06-20T00:18:10Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (861 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260620 00:18:10.227918 392 mini_oidc.cc:152] Starting JWKS server
I20260620 00:18:10.229087 392 webserver.cc:460] Webserver started at https://127.0.0.1:34429/ using document root <none> and password file <none>
I20260620 00:18:10.229218 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 00:18:10.229406 392 webserver.cc:460] Webserver started at http://127.0.0.1:43331/ using document root <none> and password file <none>
2026-06-20T00:18:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:10Z Disabled control of system clock
I20260620 00:18:10.238077 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40647
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40647
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34429/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/testchainca.cert with env {}
W20260620 00:18:10.320567 5516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:10.320807 5516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:10.320832 5516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:10.322013 5516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:10.322054 5516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:10.322067 5516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:10.322080 5516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:10.322093 5516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:18:10.322106 5516 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:18:10.322124 5516 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34429/jwks.json
I20260620 00:18:10.323655 5516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40647
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34429/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:10.323859 5516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:10.324036 5516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:10.329070 5521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:10.329061 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:10.329118 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:10.329226 5516 server_base.cc:1034] running on GCE node
I20260620 00:18:10.329762 5516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:10.330106 5516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:10.331290 5516 hybrid_clock.cc:648] HybridClock initialized: now 1781914690331279 us; error 29 us; skew 500 ppm
I20260620 00:18:10.332680 5516 webserver.cc:460] Webserver started at http://127.0.98.62:44047/ using document root <none> and password file <none>
I20260620 00:18:10.332929 5516 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:10.332999 5516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:10.333143 5516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:10.334256 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "2b1b9640c2ee458e9807793a557d0863"
format_stamp: "Formatted at 2026-06-20 00:18:10 on dist-test-slave-r81b"
server_key: "16df4a3d119f161a27df838ae6cdd13f"
server_key_iv: "2033613ade526ec3e3ffaf558e2f687f"
server_key_version: "encryptionkey@0"
I20260620 00:18:10.334635 5516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "2b1b9640c2ee458e9807793a557d0863"
format_stamp: "Formatted at 2026-06-20 00:18:10 on dist-test-slave-r81b"
server_key: "16df4a3d119f161a27df838ae6cdd13f"
server_key_iv: "2033613ade526ec3e3ffaf558e2f687f"
server_key_version: "encryptionkey@0"
I20260620 00:18:10.335987 5516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:18:10.336875 5530 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:10.337045 5516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:10.337148 5516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "2b1b9640c2ee458e9807793a557d0863"
format_stamp: "Formatted at 2026-06-20 00:18:10 on dist-test-slave-r81b"
server_key: "16df4a3d119f161a27df838ae6cdd13f"
server_key_iv: "2033613ade526ec3e3ffaf558e2f687f"
server_key_version: "encryptionkey@0"
I20260620 00:18:10.337236 5516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:10.363770 5516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:10.364114 5516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:10.364298 5516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:10.368489 5516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40647
I20260620 00:18:10.368527 5582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40647 every 8 connection(s)
I20260620 00:18:10.368888 5516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:10.369985 5583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:10.372608 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863: Bootstrap starting.
I20260620 00:18:10.373271 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 5516
I20260620 00:18:10.373293 5583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:10.373355 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:10.373561 392 external_mini_cluster.cc:1428] Setting key 3cf560173bb53c300df5a9a0cce7fb15
I20260620 00:18:10.373596 5583 log.cc:824] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:10.374480 5583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863: No bootstrap required, opened a new log
I20260620 00:18:10.377192 5583 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1b9640c2ee458e9807793a557d0863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260620 00:18:10.377359 5583 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:10.377390 5583 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b1b9640c2ee458e9807793a557d0863, State: Initialized, Role: FOLLOWER
I20260620 00:18:10.377501 5583 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1b9640c2ee458e9807793a557d0863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260620 00:18:10.377585 5583 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:10.377624 5583 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:10.377682 5583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:10.378221 5583 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1b9640c2ee458e9807793a557d0863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260620 00:18:10.378338 5583 leader_election.cc:302] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b1b9640c2ee458e9807793a557d0863; no voters:
I20260620 00:18:10.378512 5583 leader_election.cc:288] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:10.378561 5588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:10.378691 5588 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [term 1 LEADER]: Becoming Leader. State: Replica: 2b1b9640c2ee458e9807793a557d0863, State: Running, Role: LEADER
I20260620 00:18:10.378774 5588 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1b9640c2ee458e9807793a557d0863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } }
I20260620 00:18:10.378808 5583 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:10.379096 5588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b1b9640c2ee458e9807793a557d0863. Latest consensus state: current_term: 1 leader_uuid: "2b1b9640c2ee458e9807793a557d0863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1b9640c2ee458e9807793a557d0863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } } }
I20260620 00:18:10.379194 5588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:10.379104 5589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b1b9640c2ee458e9807793a557d0863" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b1b9640c2ee458e9807793a557d0863" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40647 } } }
I20260620 00:18:10.379369 5589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:10.379429 5594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:10.379925 5594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:10.380997 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.374100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48810 (local address 127.0.98.62:40647)
0620 00:18:10.374346 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:10.374355 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:10.374717 (+ 362us) server_negotiation.cc:407] Connection header received
0620 00:18:10.375728 (+ 1011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:10.375738 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:10.375817 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:10.375979 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:10.376156 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.377528 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.378409 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.379115 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.379611 (+ 496us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:10.380020 (+ 409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:10.380024 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:10.380024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:10.380258 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:10.380301 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:10.380316 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:10.380377 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:10.380678 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 00:18:10.380771 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":100,"threads_started":1}
I20260620 00:18:10.381814 5594 catalog_manager.cc:1347] Generated new cluster ID: bcb81481a4a946cb943553506adac695
I20260620 00:18:10.381865 5594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:10.381978 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:18:10.391448 5594 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:10.392242 5594 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:10.403790 5594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b1b9640c2ee458e9807793a557d0863: Generated new TSK 0
I20260620 00:18:10.404021 5594 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 00:18:10.428289 5613 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260620 00:18:10.428399 5613 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 00:18:10.428602 5587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.412026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48820 (local address 127.0.98.62:40647)
0620 00:18:10.412106 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:10.412108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:10.412288 (+ 180us) server_negotiation.cc:407] Connection header received
0620 00:18:10.412366 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:10.412367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:10.412381 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:10.412466 (+ 85us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:18:10.412645 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.413437 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.414273 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.414577 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.414659 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:10.414877 (+ 218us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:18:10.428359 (+ 13482us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 00:18:10.428515 (+ 156us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48820: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":57}
W20260620 00:18:10.428687 5587 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48820: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260620 00:18:10.429072 5612 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:10.411925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40647 (local address 127.0.0.1:48820)
0620 00:18:10.412209 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:10.412227 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:10.412287 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:10.412478 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:10.412479 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:10.412481 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:18:10.412576 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:10.412578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.413548 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.413549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:10.414173 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:10.414175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.414676 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.414680 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:10.414733 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:10.414757 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:18:10.428628 (+ 13871us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0620 00:18:10.428792 (+ 164us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40647: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":117,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:40647: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34429/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260620 00:18:10.429756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 5516
2026-06-20T00:18:10Z chronyd exiting
I20260620 00:18:10.466132 392 test_util.cc:175] -----------------------------------------------
I20260620 00:18:10.466239 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1781914666918284-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (239 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260620 00:18:10.466997 392 mini_oidc.cc:152] Starting JWKS server
I20260620 00:18:10.467694 392 webserver.cc:460] Webserver started at https://127.0.0.1:42111/ using document root <none> and password file <none>
I20260620 00:18:10.467763 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 00:18:10.467954 392 webserver.cc:460] Webserver started at http://127.0.0.1:44663/ using document root <none> and password file <none>
2026-06-20T00:18:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:10Z Disabled control of system clock
I20260620 00:18:10.475701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46597
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46597
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42111/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/testchainca.cert with env {}
W20260620 00:18:10.557039 5619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:10.557264 5619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:10.557288 5619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:10.558410 5619 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:10.558442 5619 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:10.558455 5619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:10.558468 5619 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:10.558480 5619 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:18:10.558492 5619 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:18:10.558503 5619 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42111/jwks.json
I20260620 00:18:10.559939 5619 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46597
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42111/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5619
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:10.560189 5619 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:10.560395 5619 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:10.565562 5625 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:10.565616 5619 server_base.cc:1034] running on GCE node
W20260620 00:18:10.565577 5627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:10.565577 5624 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:10.566056 5619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:10.566323 5619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:10.567575 5619 hybrid_clock.cc:648] HybridClock initialized: now 1781914690567528 us; error 31 us; skew 500 ppm
I20260620 00:18:10.569074 5619 webserver.cc:460] Webserver started at http://127.0.98.62:43229/ using document root <none> and password file <none>
I20260620 00:18:10.569324 5619 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:10.569396 5619 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:10.569547 5619 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:10.570724 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "51766b8ba92943c88e4b7891f7a77bd4"
format_stamp: "Formatted at 2026-06-20 00:18:10 on dist-test-slave-r81b"
server_key: "39c7f823208b0de1fcca75d9280cfffe"
server_key_iv: "67d70abf77a00150d0094b043db4626b"
server_key_version: "encryptionkey@0"
I20260620 00:18:10.571089 5619 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "51766b8ba92943c88e4b7891f7a77bd4"
format_stamp: "Formatted at 2026-06-20 00:18:10 on dist-test-slave-r81b"
server_key: "39c7f823208b0de1fcca75d9280cfffe"
server_key_iv: "67d70abf77a00150d0094b043db4626b"
server_key_version: "encryptionkey@0"
I20260620 00:18:10.572468 5619 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:10.573279 5633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:10.573536 5619 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:10.573661 5619 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "51766b8ba92943c88e4b7891f7a77bd4"
format_stamp: "Formatted at 2026-06-20 00:18:10 on dist-test-slave-r81b"
server_key: "39c7f823208b0de1fcca75d9280cfffe"
server_key_iv: "67d70abf77a00150d0094b043db4626b"
server_key_version: "encryptionkey@0"
I20260620 00:18:10.573742 5619 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:10.584219 5619 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:10.584581 5619 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:10.584743 5619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:10.589056 5619 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46597
I20260620 00:18:10.589087 5685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46597 every 8 connection(s)
I20260620 00:18:10.589470 5619 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:10.590119 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 5619
I20260620 00:18:10.590219 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:10.590412 392 external_mini_cluster.cc:1428] Setting key 13edd2090aa127cbd6e05ff30226d5d4
I20260620 00:18:10.590703 5686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:10.593710 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4: Bootstrap starting.
I20260620 00:18:10.594348 5686 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:10.594630 5686 log.cc:824] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:10.595327 5686 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4: No bootstrap required, opened a new log
I20260620 00:18:10.596035 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.590948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49510 (local address 127.0.98.62:46597)
0620 00:18:10.591176 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:10.591184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:10.591209 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:18:10.591791 (+ 582us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:10.591798 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:10.591864 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:10.592015 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:10.592256 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.593156 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.594015 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.594480 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.594637 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:10.595352 (+ 715us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:10.595356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:10.595357 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:10.595572 (+ 215us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:10.595604 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:10.595615 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:10.595690 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:10.595767 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:18:10.595851 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":88,"threads_started":1}
I20260620 00:18:10.597090 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:18:10.597636 5686 raft_consensus.cc:348] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51766b8ba92943c88e4b7891f7a77bd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46597 } }
I20260620 00:18:10.597820 5686 raft_consensus.cc:374] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:10.597859 5686 raft_consensus.cc:729] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51766b8ba92943c88e4b7891f7a77bd4, State: Initialized, Role: FOLLOWER
I20260620 00:18:10.597975 5686 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51766b8ba92943c88e4b7891f7a77bd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46597 } }
I20260620 00:18:10.598058 5686 raft_consensus.cc:388] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:10.598091 5686 raft_consensus.cc:482] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:10.598129 5686 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:10.598668 5686 raft_consensus.cc:504] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51766b8ba92943c88e4b7891f7a77bd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46597 } }
I20260620 00:18:10.598752 5686 leader_election.cc:302] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 51766b8ba92943c88e4b7891f7a77bd4; no voters:
I20260620 00:18:10.598872 5686 leader_election.cc:288] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:10.598953 5691 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:10.599159 5686 sys_catalog.cc:564] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:10.599159 5691 raft_consensus.cc:686] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [term 1 LEADER]: Becoming Leader. State: Replica: 51766b8ba92943c88e4b7891f7a77bd4, State: Running, Role: LEADER
I20260620 00:18:10.599308 5691 consensus_queue.cc:237] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51766b8ba92943c88e4b7891f7a77bd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46597 } }
I20260620 00:18:10.599656 5693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 51766b8ba92943c88e4b7891f7a77bd4. Latest consensus state: current_term: 1 leader_uuid: "51766b8ba92943c88e4b7891f7a77bd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51766b8ba92943c88e4b7891f7a77bd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46597 } } }
I20260620 00:18:10.599750 5693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:10.599653 5692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "51766b8ba92943c88e4b7891f7a77bd4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "51766b8ba92943c88e4b7891f7a77bd4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46597 } } }
I20260620 00:18:10.599923 5692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:10.600447 5700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:10.600948 5700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:10.602494 5700 catalog_manager.cc:1347] Generated new cluster ID: a3647c0385ba406e96292a33a399d7ae
I20260620 00:18:10.602579 5700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:10.608215 5700 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:10.608877 5700 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:10.624575 5700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 51766b8ba92943c88e4b7891f7a77bd4: Generated new TSK 0
I20260620 00:18:10.624840 5700 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260620 00:18:10.655045 5716 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260620 00:18:10.655171 5716 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260620 00:18:10.655438 5689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.639107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49514 (local address 127.0.98.62:46597)
0620 00:18:10.639159 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:10.639162 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:10.639257 (+ 95us) server_negotiation.cc:407] Connection header received
0620 00:18:10.639309 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:10.639310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:10.639323 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:10.639414 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:18:10.639543 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.640383 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.641441 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.641682 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.641721 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:10.641991 (+ 270us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:18:10.655209 (+ 13218us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 00:18:10.655363 (+ 154us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49514: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":28}
W20260620 00:18:10.655499 5689 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49514: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260620 00:18:10.655535 5715 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:10.639024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46597 (local address 127.0.0.1:49514)
0620 00:18:10.639174 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:10.639187 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:10.639253 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:10.639417 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:10.639418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:10.639419 (+ 1us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:18:10.639480 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:10.639482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.640497 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.640499 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:10.641263 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:10.641266 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.641784 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.641788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:10.641873 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:10.641878 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:18:10.655338 (+ 13460us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0620 00:18:10.655437 (+ 99us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:46597: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:42111/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":103,"threads_started":1}
I20260620 00:18:10.655942 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 5619
2026-06-20T00:18:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260620 00:18:10.693663 392 mini_oidc.cc:152] Starting JWKS server
I20260620 00:18:10.694322 392 webserver.cc:460] Webserver started at https://127.0.0.1:34925/ using document root <none> and password file <none>
I20260620 00:18:10.694370 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260620 00:18:10.694507 392 webserver.cc:460] Webserver started at http://127.0.0.1:42345/ using document root <none> and password file <none>
2026-06-20T00:18:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:10Z Disabled control of system clock
I20260620 00:18:10.702391 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40265
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:36897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40265
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34925/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 00:18:10.786942 5722 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:10.787216 5722 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:10.787256 5722 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:10.788695 5722 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:10.788770 5722 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:10.788793 5722 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:10.788818 5722 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:10.788853 5722 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:18:10.788880 5722 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:18:10.788914 5722 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34925/jwks.json
I20260620 00:18:10.790516 5722 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40265
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34925/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5722
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:10.790882 5722 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:10.791260 5722 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:10.796568 5728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:10.796648 5727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:10.796561 5730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:10.796846 5722 server_base.cc:1034] running on GCE node
I20260620 00:18:10.797171 5722 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:10.797477 5722 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:10.798648 5722 hybrid_clock.cc:648] HybridClock initialized: now 1781914690798636 us; error 33 us; skew 500 ppm
I20260620 00:18:10.799961 5722 webserver.cc:460] Webserver started at http://127.0.98.62:40219/ using document root <none> and password file <none>
I20260620 00:18:10.800278 5722 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:10.800356 5722 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:10.800509 5722 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:10.801640 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "86163a61c3b14e9e9bd21d3944f80213"
format_stamp: "Formatted at 2026-06-20 00:18:10 on dist-test-slave-r81b"
server_key: "7e325c215900e64fda32b4385dd11134"
server_key_iv: "1dc0b88d16c641ab776f3f673426fd23"
server_key_version: "encryptionkey@0"
I20260620 00:18:10.802011 5722 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "86163a61c3b14e9e9bd21d3944f80213"
format_stamp: "Formatted at 2026-06-20 00:18:10 on dist-test-slave-r81b"
server_key: "7e325c215900e64fda32b4385dd11134"
server_key_iv: "1dc0b88d16c641ab776f3f673426fd23"
server_key_version: "encryptionkey@0"
I20260620 00:18:10.803390 5722 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:10.804203 5736 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:10.804391 5722 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:10.804495 5722 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "86163a61c3b14e9e9bd21d3944f80213"
format_stamp: "Formatted at 2026-06-20 00:18:10 on dist-test-slave-r81b"
server_key: "7e325c215900e64fda32b4385dd11134"
server_key_iv: "1dc0b88d16c641ab776f3f673426fd23"
server_key_version: "encryptionkey@0"
I20260620 00:18:10.804586 5722 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:10.823335 5722 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:10.823699 5722 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:10.823871 5722 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:10.828385 5722 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40265
I20260620 00:18:10.828467 5788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40265 every 8 connection(s)
I20260620 00:18:10.828850 5722 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:10.829571 5789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:10.832518 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213: Bootstrap starting.
I20260620 00:18:10.833119 5789 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:10.833386 5789 log.cc:824] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:10.834116 5789 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213: No bootstrap required, opened a new log
I20260620 00:18:10.835825 5789 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86163a61c3b14e9e9bd21d3944f80213" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } }
I20260620 00:18:10.835990 5789 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:10.836016 5789 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86163a61c3b14e9e9bd21d3944f80213, State: Initialized, Role: FOLLOWER
I20260620 00:18:10.836190 5789 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86163a61c3b14e9e9bd21d3944f80213" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } }
I20260620 00:18:10.836283 5789 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:10.836309 5789 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:10.836341 5789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:10.836876 5789 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86163a61c3b14e9e9bd21d3944f80213" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } }
I20260620 00:18:10.836973 5789 leader_election.cc:302] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86163a61c3b14e9e9bd21d3944f80213; no voters:
I20260620 00:18:10.837109 5789 leader_election.cc:288] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:10.837198 5792 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:10.837299 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 5722
I20260620 00:18:10.837423 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:10.837414 5792 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [term 1 LEADER]: Becoming Leader. State: Replica: 86163a61c3b14e9e9bd21d3944f80213, State: Running, Role: LEADER
I20260620 00:18:10.837416 5789 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:10.837675 392 external_mini_cluster.cc:1428] Setting key 5418760b732acc65f0189e1277fb3b1e
I20260620 00:18:10.837884 5792 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86163a61c3b14e9e9bd21d3944f80213" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } }
I20260620 00:18:10.838634 5794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86163a61c3b14e9e9bd21d3944f80213. Latest consensus state: current_term: 1 leader_uuid: "86163a61c3b14e9e9bd21d3944f80213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86163a61c3b14e9e9bd21d3944f80213" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } } }
I20260620 00:18:10.838735 5794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:10.839022 5793 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86163a61c3b14e9e9bd21d3944f80213" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86163a61c3b14e9e9bd21d3944f80213" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40265 } } }
I20260620 00:18:10.839166 5793 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:10.839392 5803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:10.840005 5803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:10.841894 5803 catalog_manager.cc:1347] Generated new cluster ID: ae5ec4fdc6ae444cab1b74a6a958a95c
I20260620 00:18:10.841993 5803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:10.844497 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.838221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34216 (local address 127.0.98.62:40265)
0620 00:18:10.838501 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:10.838513 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:10.838956 (+ 443us) server_negotiation.cc:407] Connection header received
0620 00:18:10.839117 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:10.839127 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:10.839219 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:10.839687 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:10.839767 (+ 80us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.840993 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.842116 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.842753 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.842904 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:10.843684 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:10.843690 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:10.843690 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:10.843929 (+ 239us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:10.843980 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:10.843996 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:10.844078 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:10.844174 (+ 96us) server_negotiation.cc:299] Negotiation successful
0620 00:18:10.844261 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
I20260620 00:18:10.845474 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:18:10.859354 5803 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:10.860129 5803 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:10.871004 5803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86163a61c3b14e9e9bd21d3944f80213: Generated new TSK 0
I20260620 00:18:10.871256 5803 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:18:10.886140 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:10.882057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34230 (local address 127.0.98.62:40265)
0620 00:18:10.882126 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:10.882128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:10.882258 (+ 130us) server_negotiation.cc:407] Connection header received
0620 00:18:10.882312 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:10.882314 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:10.882329 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:10.882433 (+ 104us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:18:10.882560 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.883347 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.884275 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.884597 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.884678 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:10.884843 (+ 165us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:18:10.885895 (+ 1052us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34925/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 00:18:10.886043 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34925/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 00:18:10.886209 5799 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34230: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34925/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260620 00:18:10.886282 5818 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:10.881950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40265 (local address 127.0.0.1:34230)
0620 00:18:10.882176 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:10.882188 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:10.882254 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:10.882437 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:10.882437 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:10.882439 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:18:10.882492 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:10.882494 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.883486 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.883487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:10.884120 (+ 633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:10.884122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:10.884682 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:10.884686 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:10.884729 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:10.884735 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:18:10.886011 (+ 1276us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34925/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0620 00:18:10.886165 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40265: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34925/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":202,"thread_start_us":158,"threads_started":1}
I20260620 00:18:10.886749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 5722
E20260620 00:18:10.897132 5819 webserver.cc:533] Webserver: sslize failed:
E20260620 00:18:10.897246 5819 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-06-20T00:18:10Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (230 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-06-20T00:18:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:10Z Disabled control of system clock
I20260620 00:18:10.932281 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37405
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45953
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37405
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/testchainca.cert with env {}
W20260620 00:18:11.013146 5823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:11.013371 5823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:11.013394 5823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:11.014541 5823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:11.014575 5823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:11.014588 5823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:11.014601 5823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:11.014613 5823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260620 00:18:11.014626 5823 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260620 00:18:11.014636 5823 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260620 00:18:11.016070 5823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45953
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37405
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:11.016297 5823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:11.016487 5823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:11.021229 5828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:11.021265 5831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:11.021296 5829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:11.021727 5823 server_base.cc:1034] running on GCE node
I20260620 00:18:11.022044 5823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:11.022362 5823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:11.023556 5823 hybrid_clock.cc:648] HybridClock initialized: now 1781914691023542 us; error 32 us; skew 500 ppm
I20260620 00:18:11.025022 5823 webserver.cc:460] Webserver started at http://127.0.98.62:33315/ using document root <none> and password file <none>
I20260620 00:18:11.025331 5823 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:11.025382 5823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:11.025520 5823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:11.026638 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "ef50754dae07479788bd50066bdc0287"
format_stamp: "Formatted at 2026-06-20 00:18:11 on dist-test-slave-r81b"
server_key: "38803f094875b9e626cb17eb2f5fc11a"
server_key_iv: "f54e5b56abdf476beb945e43c3dc8225"
server_key_version: "encryptionkey@0"
I20260620 00:18:11.027004 5823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "ef50754dae07479788bd50066bdc0287"
format_stamp: "Formatted at 2026-06-20 00:18:11 on dist-test-slave-r81b"
server_key: "38803f094875b9e626cb17eb2f5fc11a"
server_key_iv: "f54e5b56abdf476beb945e43c3dc8225"
server_key_version: "encryptionkey@0"
I20260620 00:18:11.028532 5823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:11.029351 5837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:11.029594 5823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:11.029693 5823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "ef50754dae07479788bd50066bdc0287"
format_stamp: "Formatted at 2026-06-20 00:18:11 on dist-test-slave-r81b"
server_key: "38803f094875b9e626cb17eb2f5fc11a"
server_key_iv: "f54e5b56abdf476beb945e43c3dc8225"
server_key_version: "encryptionkey@0"
I20260620 00:18:11.029770 5823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:11.043320 5823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:11.043689 5823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:11.043848 5823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:11.048290 5823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37405
I20260620 00:18:11.048343 5889 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37405 every 8 connection(s)
I20260620 00:18:11.048715 5823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:11.049757 5890 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:11.052094 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287: Bootstrap starting.
I20260620 00:18:11.052750 5890 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:11.053048 5890 log.cc:824] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:11.053896 5890 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287: No bootstrap required, opened a new log
I20260620 00:18:11.055564 5890 raft_consensus.cc:348] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef50754dae07479788bd50066bdc0287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } }
I20260620 00:18:11.055708 5890 raft_consensus.cc:374] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:11.055732 5890 raft_consensus.cc:729] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef50754dae07479788bd50066bdc0287, State: Initialized, Role: FOLLOWER
I20260620 00:18:11.055899 5890 consensus_queue.cc:260] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef50754dae07479788bd50066bdc0287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } }
I20260620 00:18:11.055964 5890 raft_consensus.cc:388] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:11.056025 5890 raft_consensus.cc:482] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:11.056077 5890 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:11.056715 5890 raft_consensus.cc:504] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef50754dae07479788bd50066bdc0287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } }
I20260620 00:18:11.056736 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 5823
I20260620 00:18:11.056800 5890 leader_election.cc:302] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ef50754dae07479788bd50066bdc0287; no voters:
I20260620 00:18:11.056844 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:11.056996 5890 leader_election.cc:288] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:11.057025 5893 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:11.057091 392 external_mini_cluster.cc:1428] Setting key 12aa1523625f93cc0ce13dc10575eb30
I20260620 00:18:11.057145 5893 raft_consensus.cc:686] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [term 1 LEADER]: Becoming Leader. State: Replica: ef50754dae07479788bd50066bdc0287, State: Running, Role: LEADER
I20260620 00:18:11.057263 5890 sys_catalog.cc:564] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:11.057266 5893 consensus_queue.cc:237] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef50754dae07479788bd50066bdc0287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } }
I20260620 00:18:11.057844 5895 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ef50754dae07479788bd50066bdc0287. Latest consensus state: current_term: 1 leader_uuid: "ef50754dae07479788bd50066bdc0287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef50754dae07479788bd50066bdc0287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } } }
I20260620 00:18:11.057945 5895 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:11.058044 5894 sys_catalog.cc:455] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ef50754dae07479788bd50066bdc0287" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ef50754dae07479788bd50066bdc0287" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37405 } } }
I20260620 00:18:11.058173 5894 sys_catalog.cc:458] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:11.058507 5904 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:11.059098 5904 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:11.060741 5904 catalog_manager.cc:1347] Generated new cluster ID: 6b8b47a292e14b52bd8a955b5fdae563
I20260620 00:18:11.060799 5904 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:11.063217 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:11.057698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.0.98.62:37405)
0620 00:18:11.058247 (+ 549us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:11.058262 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:11.058304 (+ 42us) server_negotiation.cc:407] Connection header received
0620 00:18:11.058457 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:11.058466 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:11.058554 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:11.059022 (+ 468us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:11.059111 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:11.060349 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:11.061278 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:11.061709 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:11.061846 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:11.062555 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:11.062559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:11.062560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:11.062742 (+ 182us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:11.062775 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:11.062786 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:11.062862 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:11.062938 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 00:18:11.063028 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":59,"threads_started":1}
I20260620 00:18:11.065220 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:18:11.078899 5904 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:11.079473 5904 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:11.086513 5904 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ef50754dae07479788bd50066bdc0287: Generated new TSK 0
I20260620 00:18:11.086756 5904 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:18:11.145087 5899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:11.095729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51104 (local address 127.0.98.62:37405)
0620 00:18:11.095807 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:11.095809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:11.095921 (+ 112us) server_negotiation.cc:407] Connection header received
0620 00:18:11.096018 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:11.096018 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:11.096031 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:11.096121 (+ 90us) server_negotiation.cc:226] Negotiated authn=JWT
0620 00:18:11.096298 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:11.096812 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:11.097638 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:11.097879 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:11.097954 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:11.098171 (+ 217us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0620 00:18:11.144746 (+ 46575us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 00:18:11.144989 (+ 243us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:51104: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":42}
W20260620 00:18:11.145176 5899 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:51104: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260620 00:18:11.145200 5919 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:11.095660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37405 (local address 127.0.0.1:51104)
0620 00:18:11.095846 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:11.095855 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:11.095939 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:11.096121 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:11.096122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:11.096124 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0620 00:18:11.096212 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:11.096215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:11.096913 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:11.096914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:11.097538 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:11.097540 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:11.097958 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:11.097962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:11.098037 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:11.098042 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0620 00:18:11.144926 (+ 46884us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0620 00:18:11.145073 (+ 147us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:37405: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":106,"threads_started":1}
I20260620 00:18:11.145432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 5823
2026-06-20T00:18:11Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (238 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (91 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:11 dist-test-slave-r81b krb5kdc[5931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:11 dist-test-slave-r81b krb5kdc[5931](info): set up 2 sockets
Jun 20 00:18:11 dist-test-slave-r81b krb5kdc[5931](info): commencing operation
krb5kdc: starting...
W20260620 00:18:12.360502 392 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:12 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:18:12.479759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37857
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37857
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:12.560731 5944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:12.560992 5944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:12.561029 5944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:12.562204 5944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:12.562283 5944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:12.562309 5944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:12.562331 5944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:12.562361 5944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:12.563947 5944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37857
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5944
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:12.564268 5944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:12.564503 5944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:12.569469 5950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:12.569531 5949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:12.569571 5952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:12.569641 5944 server_base.cc:1034] running on GCE node
I20260620 00:18:12.569986 5944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:12.570289 5944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:12.571461 5944 hybrid_clock.cc:648] HybridClock initialized: now 1781914692571446 us; error 46 us; skew 500 ppm
Jun 20 00:18:12 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:12.574038 5944 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:18:12.574515 5944 webserver.cc:460] Webserver started at http://127.0.98.62:42517/ using document root <none> and password file <none>
I20260620 00:18:12.574818 5944 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:12.574887 5944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:12.575052 5944 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:12.576246 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "097b7160ef544ba5b5bc92c75f243f57"
format_stamp: "Formatted at 2026-06-20 00:18:12 on dist-test-slave-r81b"
server_key: "020fc6ab1aa957d8b7001ab58e5e024e"
server_key_iv: "15977ab0809c3959abe352f5aac7cbf2"
server_key_version: "encryptionkey@0"
I20260620 00:18:12.576609 5944 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "097b7160ef544ba5b5bc92c75f243f57"
format_stamp: "Formatted at 2026-06-20 00:18:12 on dist-test-slave-r81b"
server_key: "020fc6ab1aa957d8b7001ab58e5e024e"
server_key_iv: "15977ab0809c3959abe352f5aac7cbf2"
server_key_version: "encryptionkey@0"
I20260620 00:18:12.577958 5944 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:12.578720 5959 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:12.578888 5944 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:12.579016 5944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "097b7160ef544ba5b5bc92c75f243f57"
format_stamp: "Formatted at 2026-06-20 00:18:12 on dist-test-slave-r81b"
server_key: "020fc6ab1aa957d8b7001ab58e5e024e"
server_key_iv: "15977ab0809c3959abe352f5aac7cbf2"
server_key_version: "encryptionkey@0"
I20260620 00:18:12.579102 5944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:12.593616 5944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:12.596175 5944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:12.596359 5944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:12.600322 5944 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37857
I20260620 00:18:12.600380 6011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37857 every 8 connection(s)
I20260620 00:18:12.600929 5944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:12.601629 6012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:12.603876 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57: Bootstrap starting.
I20260620 00:18:12.604525 6012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:12.604782 6012 log.cc:824] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:12.604862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 5944
I20260620 00:18:12.604982 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:12.605197 392 external_mini_cluster.cc:1428] Setting key 2825ec8130837df29d2a309fa4742864
I20260620 00:18:12.605597 6012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57: No bootstrap required, opened a new log
I20260620 00:18:12.608572 6012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097b7160ef544ba5b5bc92c75f243f57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260620 00:18:12.608755 6012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:12.608790 6012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 097b7160ef544ba5b5bc92c75f243f57, State: Initialized, Role: FOLLOWER
I20260620 00:18:12.608916 6012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097b7160ef544ba5b5bc92c75f243f57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260620 00:18:12.608987 6012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:12.609014 6012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:12.609050 6012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:12.609754 6012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097b7160ef544ba5b5bc92c75f243f57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260620 00:18:12.609848 6012 leader_election.cc:302] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 097b7160ef544ba5b5bc92c75f243f57; no voters:
I20260620 00:18:12.610064 6012 leader_election.cc:288] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:12.610123 6017 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:12.610280 6017 raft_consensus.cc:686] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [term 1 LEADER]: Becoming Leader. State: Replica: 097b7160ef544ba5b5bc92c75f243f57, State: Running, Role: LEADER
I20260620 00:18:12.610351 6012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:12.610433 6017 consensus_queue.cc:237] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097b7160ef544ba5b5bc92c75f243f57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } }
I20260620 00:18:12.610778 6017 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 097b7160ef544ba5b5bc92c75f243f57. Latest consensus state: current_term: 1 leader_uuid: "097b7160ef544ba5b5bc92c75f243f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097b7160ef544ba5b5bc92c75f243f57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } } }
I20260620 00:18:12.610898 6017 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:12.611181 6025 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:12.612107 6025 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:12.612224 6018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "097b7160ef544ba5b5bc92c75f243f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "097b7160ef544ba5b5bc92c75f243f57" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37857 } } }
I20260620 00:18:12.612334 6018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57 [sys.catalog]: This master's current role is: LEADER
Jun 20 00:18:12 dist-test-slave-r81b krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914692, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:12.614097 6025 catalog_manager.cc:1347] Generated new cluster ID: d9b132f3806a42a2a3fa2657c2b14c56
I20260620 00:18:12.614683 6025 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:12.618108 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:12.605739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.0.98.62:37857)
0620 00:18:12.605960 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:12.605969 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:12.606005 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:18:12.606895 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:12.606906 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:12.606981 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:12.607138 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:12.607819 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.609292 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.610210 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.610943 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.611053 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:12.614083 (+ 3030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:12.614087 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:12.614088 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:12.614100 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:12.616176 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:12.616614 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:12.616617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:12.616617 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:12.616719 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:12.616920 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:12.616920 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:12.616921 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:12.617209 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:12.617352 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:12.617779 (+ 427us) server_negotiation.cc:299] Negotiation successful
0620 00:18:12.617906 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260620 00:18:12.638015 6025 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:12.638656 6025 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:12.641548 6025 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 097b7160ef544ba5b5bc92c75f243f57: Generated new TSK 0
I20260620 00:18:12.641768 6025 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:18:12.670662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:12.756644 6038 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:12.756889 6038 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:12.756915 6038 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:12.758118 6038 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:12.758162 6038 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:12.758190 6038 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:12.759684 6038 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6038
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:12.759933 6038 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:12.760234 6038 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:12.765614 6043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:12.765620 6044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:12.765760 6038 server_base.cc:1034] running on GCE node
W20260620 00:18:12.765851 6046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:12.766222 6038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:12.766518 6038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:12.767729 6038 hybrid_clock.cc:648] HybridClock initialized: now 1781914692767685 us; error 56 us; skew 500 ppm
Jun 20 00:18:12 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:12.770669 6038 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:18:12.771148 6038 webserver.cc:460] Webserver started at http://127.0.98.1:41697/ using document root <none> and password file <none>
I20260620 00:18:12.771381 6038 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:12.771451 6038 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:12.771598 6038 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:12.772751 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "c7c59cfca046473a9ad7a1dd4acb8540"
format_stamp: "Formatted at 2026-06-20 00:18:12 on dist-test-slave-r81b"
server_key: "178fc55cc6d6fd0b6ff217b5fbdcc449"
server_key_iv: "a4a7bc5ef44372c00d63f2d8add4d6b5"
server_key_version: "encryptionkey@0"
I20260620 00:18:12.773118 6038 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c7c59cfca046473a9ad7a1dd4acb8540"
format_stamp: "Formatted at 2026-06-20 00:18:12 on dist-test-slave-r81b"
server_key: "178fc55cc6d6fd0b6ff217b5fbdcc449"
server_key_iv: "a4a7bc5ef44372c00d63f2d8add4d6b5"
server_key_version: "encryptionkey@0"
I20260620 00:18:12.774498 6038 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:12.775345 6053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:12.775501 6038 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:12.775604 6038 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "c7c59cfca046473a9ad7a1dd4acb8540"
format_stamp: "Formatted at 2026-06-20 00:18:12 on dist-test-slave-r81b"
server_key: "178fc55cc6d6fd0b6ff217b5fbdcc449"
server_key_iv: "a4a7bc5ef44372c00d63f2d8add4d6b5"
server_key_version: "encryptionkey@0"
I20260620 00:18:12.775684 6038 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:12.794132 6038 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:12.796602 6038 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:12.796764 6038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:12.797017 6038 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:12.797325 6038 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:12.797375 6038 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:12.797412 6038 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:12.797457 6038 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:12.803259 6038 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35657
I20260620 00:18:12.803304 6166 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35657 every 8 connection(s)
I20260620 00:18:12.803632 6038 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:18:12.806241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 6038
I20260620 00:18:12.806353 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:12.806546 392 external_mini_cluster.cc:1428] Setting key 3da5ef76ecfcd72145d83d9fd1f6ee63
Jun 20 00:18:12 dist-test-slave-r81b krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:18:12.815423 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:12.804384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60737 (local address 127.0.98.62:37857)
0620 00:18:12.804488 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:12.804491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:12.804666 (+ 175us) server_negotiation.cc:407] Connection header received
0620 00:18:12.805745 (+ 1079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:12.805747 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:12.805760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:12.805819 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:12.806825 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.807514 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.809035 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.809367 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.809446 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:12.811945 (+ 2499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:12.811961 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:12.811962 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:12.811970 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:12.813923 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:12.814348 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:12.814349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:12.814350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:12.814406 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:12.814671 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:12.814672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:12.814672 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:12.814800 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:12.814873 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:12.815232 (+ 359us) server_negotiation.cc:299] Negotiation successful
0620 00:18:12.815322 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 00:18:12.815642 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:12.804306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37857 (local address 127.0.98.1:60737)
0620 00:18:12.804524 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:12.804578 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:12.805622 (+ 1044us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:12.805908 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:12.805912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:12.806342 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:12.806644 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:12.806651 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.807616 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.807617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:12.808938 (+ 1321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:12.808941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.809455 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.809456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:12.809561 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:12.810165 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:12.810173 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:12.811840 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:12.814014 (+ 2174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:12.814022 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:12.814025 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:12.814267 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:12.814482 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:12.814483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:12.814483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:12.814600 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:12.814878 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:12.814884 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:12.815079 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 00:18:12.815212 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 00:18:12.815384 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":87,"threads_started":1}
I20260620 00:18:12.816490 6167 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37857
I20260620 00:18:12.816602 6167 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:12.816833 6167 heartbeater.cc:511] Master 127.0.98.62:37857 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:18:12.817538 5976 ts_manager.cc:194] Registered new tserver with Master: c7c59cfca046473a9ad7a1dd4acb8540 (127.0.98.1:35657)
I20260620 00:18:12.818195 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60737
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:18:12.835942 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:12.928580 6172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:12.928834 6172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:12.928877 6172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:12.930100 6172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:12.930178 6172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:12.930226 6172 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:12.931779 6172 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:12.932097 6172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:12.932382 6172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:12.937776 6180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:12.937777 6177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:12.937841 6172 server_base.cc:1034] running on GCE node
W20260620 00:18:12.937777 6178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:12.938396 6172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:12.938669 6172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:12.939868 6172 hybrid_clock.cc:648] HybridClock initialized: now 1781914692939848 us; error 39 us; skew 500 ppm
Jun 20 00:18:12 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:12.942592 6172 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:18:12.943049 6172 webserver.cc:460] Webserver started at http://127.0.98.2:42629/ using document root <none> and password file <none>
I20260620 00:18:12.943288 6172 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:12.943363 6172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:12.943527 6172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:12.944670 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "575c7d4962d142709c90b4527f15ecea"
format_stamp: "Formatted at 2026-06-20 00:18:12 on dist-test-slave-r81b"
server_key: "6b9d557a785c7fcd21c72c128c8c6f8b"
server_key_iv: "9975a1eba59ef6da522aa14ac1416357"
server_key_version: "encryptionkey@0"
I20260620 00:18:12.945034 6172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "575c7d4962d142709c90b4527f15ecea"
format_stamp: "Formatted at 2026-06-20 00:18:12 on dist-test-slave-r81b"
server_key: "6b9d557a785c7fcd21c72c128c8c6f8b"
server_key_iv: "9975a1eba59ef6da522aa14ac1416357"
server_key_version: "encryptionkey@0"
I20260620 00:18:12.946774 6172 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 00:18:12.947558 6187 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:12.947803 6172 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:12.947893 6172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "575c7d4962d142709c90b4527f15ecea"
format_stamp: "Formatted at 2026-06-20 00:18:12 on dist-test-slave-r81b"
server_key: "6b9d557a785c7fcd21c72c128c8c6f8b"
server_key_iv: "9975a1eba59ef6da522aa14ac1416357"
server_key_version: "encryptionkey@0"
I20260620 00:18:12.947971 6172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:12.965502 6172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:12.967756 6172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:12.967912 6172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:12.968199 6172 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:12.968542 6172 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:12.968581 6172 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:12.968604 6172 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:12.968652 6172 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:12.975044 6172 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43447
I20260620 00:18:12.975112 6300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43447 every 8 connection(s)
I20260620 00:18:12.975457 6172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:12.981313 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 6172
I20260620 00:18:12.981431 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:12.981663 392 external_mini_cluster.cc:1428] Setting key 41b77f50527655e70bed0638a6a645a1
Jun 20 00:18:12 dist-test-slave-r81b krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914692, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:12.985714 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:12.976248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43895 (local address 127.0.98.62:37857)
0620 00:18:12.976325 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:12.976327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:12.976574 (+ 247us) server_negotiation.cc:407] Connection header received
0620 00:18:12.977584 (+ 1010us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:12.977585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:12.977595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:12.977659 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:12.978296 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.978851 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.979899 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.980078 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.980172 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:12.982582 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:12.982584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:12.982584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:12.982594 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:12.984245 (+ 1651us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:12.984679 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:12.984680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:12.984681 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:12.984743 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:12.985002 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:12.985003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:12.985004 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:12.985135 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:12.985225 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:12.985538 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 00:18:12.985619 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:18:12.985924 6303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:12.976278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37857 (local address 127.0.98.2:43895)
0620 00:18:12.976447 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:12.976486 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:12.977462 (+ 976us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:12.977719 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:12.977723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:12.978019 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:12.978215 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:12.978219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.978927 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.978927 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:12.979829 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:12.979831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:12.980188 (+ 357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:12.980189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:12.980310 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:12.980896 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:12.980908 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:12.982485 (+ 1577us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:12.984342 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:12.984349 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:12.984351 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:12.984588 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:12.984811 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:12.984811 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:12.984812 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:12.984932 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:12.985218 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:12.985224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:12.985379 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 00:18:12.985512 (+ 133us) client_negotiation.cc:240] Negotiation successful
0620 00:18:12.985677 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":86,"threads_started":1}
I20260620 00:18:12.986424 6301 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37857
I20260620 00:18:12.986541 6301 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:12.986749 6301 heartbeater.cc:511] Master 127.0.98.62:37857 requested a full tablet report, sending...
I20260620 00:18:12.987131 5976 ts_manager.cc:194] Registered new tserver with Master: 575c7d4962d142709c90b4527f15ecea (127.0.98.2:43447)
I20260620 00:18:12.987519 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43895
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:18:13.007875 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45557
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:13.093796 6306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:13.094022 6306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:13.094046 6306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:13.095270 6306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:13.095319 6306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:13.095348 6306 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:13.096989 6306 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45557
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37857
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:13.097254 6306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:13.097492 6306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:13.102694 6311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:13.102696 6314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:13.102797 6312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:13.103508 6306 server_base.cc:1034] running on GCE node
I20260620 00:18:13.103749 6306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:13.104046 6306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:13.105193 6306 hybrid_clock.cc:648] HybridClock initialized: now 1781914693105169 us; error 47 us; skew 500 ppm
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:13.107750 6306 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:18:13.108217 6306 webserver.cc:460] Webserver started at http://127.0.98.3:41189/ using document root <none> and password file <none>
I20260620 00:18:13.108462 6306 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:13.108532 6306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:13.108685 6306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:13.109782 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "38267432b71844159de037eca8a34ba2"
format_stamp: "Formatted at 2026-06-20 00:18:13 on dist-test-slave-r81b"
server_key: "6885933eea02672609f1c43b99808ef7"
server_key_iv: "d25e8fb2d31daa182d014618ffae0bbd"
server_key_version: "encryptionkey@0"
I20260620 00:18:13.110219 6306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "38267432b71844159de037eca8a34ba2"
format_stamp: "Formatted at 2026-06-20 00:18:13 on dist-test-slave-r81b"
server_key: "6885933eea02672609f1c43b99808ef7"
server_key_iv: "d25e8fb2d31daa182d014618ffae0bbd"
server_key_version: "encryptionkey@0"
I20260620 00:18:13.111994 6306 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 00:18:13.113039 6321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:13.113278 6306 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:13.113392 6306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "38267432b71844159de037eca8a34ba2"
format_stamp: "Formatted at 2026-06-20 00:18:13 on dist-test-slave-r81b"
server_key: "6885933eea02672609f1c43b99808ef7"
server_key_iv: "d25e8fb2d31daa182d014618ffae0bbd"
server_key_version: "encryptionkey@0"
I20260620 00:18:13.113487 6306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:13.128563 6306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:13.130858 6306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:13.131034 6306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:13.131282 6306 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:13.131594 6306 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:13.131652 6306 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:13.131699 6306 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:13.131722 6306 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:13.138109 6306 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36973
I20260620 00:18:13.138209 6434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36973 every 8 connection(s)
I20260620 00:18:13.138558 6306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:13.143625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 6306
I20260620 00:18:13.143738 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:13.143980 392 external_mini_cluster.cc:1428] Setting key 42afb914c0284d0c23dbee11b3aaa4dd
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:13.152122 6016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:13.139461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56143 (local address 127.0.98.62:37857)
0620 00:18:13.139523 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:13.139525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:13.139805 (+ 280us) server_negotiation.cc:407] Connection header received
0620 00:18:13.141085 (+ 1280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:13.141086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:13.141098 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:13.141183 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:13.142252 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.143144 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.144797 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.145131 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.145178 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:13.148098 (+ 2920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:13.148101 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:13.148102 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:13.148112 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:13.150127 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:13.150795 (+ 668us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:13.150796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:13.150797 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:13.150869 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:13.151245 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:13.151246 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:13.151247 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:13.151436 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:13.151513 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:13.151906 (+ 393us) server_negotiation.cc:299] Negotiation successful
0620 00:18:13.152027 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 00:18:13.152762 6437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:13.139431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37857 (local address 127.0.98.3:56143)
0620 00:18:13.139594 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:13.139666 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:13.140972 (+ 1306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:13.141270 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:13.141275 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:13.141762 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:13.142125 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:13.142132 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.143244 (+ 1112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.143246 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:13.144685 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:13.144688 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.145271 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.145272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:13.145438 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:13.146102 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:13.146112 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:13.147963 (+ 1851us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:13.150256 (+ 2293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:13.150267 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:13.150270 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:13.150670 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:13.150965 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:13.150966 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:13.150967 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:13.151162 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:13.151510 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:13.151530 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:13.151743 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 00:18:13.151885 (+ 142us) client_negotiation.cc:240] Negotiation successful
0620 00:18:13.152455 (+ 570us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260620 00:18:13.153322 6435 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37857
I20260620 00:18:13.153460 6435 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:13.153694 6435 heartbeater.cc:511] Master 127.0.98.62:37857 requested a full tablet report, sending...
I20260620 00:18:13.154088 5976 ts_manager.cc:194] Registered new tserver with Master: 38267432b71844159de037eca8a34ba2 (127.0.98.3:36973)
I20260620 00:18:13.154644 5976 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:56143
I20260620 00:18:13.155632 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/test-admin.keytab.
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:13.173543 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260620 00:18:13.175349 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5cc' of type FILE
W20260620 00:18:13.175374 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:13.174437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40058 (local address 127.0.98.1:35657)
0620 00:18:13.174649 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:13.174653 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:13.174664 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:13.174708 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:13.174711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:13.174724 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:13.174794 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:13.175156 (+ 362us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40058: BlockingRecv error: recv got EOF from 127.0.0.1:40058 (error 108)
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":106,"threads_started":1}
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:13.176455 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:13.177387 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:18:13.185953 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:13.178009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40068 (local address 127.0.98.1:35657)
0620 00:18:13.178088 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:13.178090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:13.178174 (+ 84us) server_negotiation.cc:407] Connection header received
0620 00:18:13.178286 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:13.178287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:13.178299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:13.178346 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:13.178792 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.179567 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.180445 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.180790 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.180868 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:13.183032 (+ 2164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:13.183040 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:13.183040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:13.183048 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:13.184783 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:13.185150 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:13.185154 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:13.185154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:13.185216 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:13.185413 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:13.185414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:13.185414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:13.185548 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:13.185708 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:13.185759 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 00:18:13.185838 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
W20260620 00:18:13.187837 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:13.187246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.0.98.1:35657)
0620 00:18:13.187304 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:13.187305 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:13.187393 (+ 88us) server_negotiation.cc:407] Connection header received
0620 00:18:13.187464 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:13.187464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:13.187475 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:13.187519 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:13.187789 (+ 270us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40078: BlockingRecv error: recv got EOF from 127.0.0.1:40078 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260620 00:18:13.188099 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:13.189128 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:13.190011 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:18:13.197540 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:13.190560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40080 (local address 127.0.98.1:35657)
0620 00:18:13.190609 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:13.190611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:13.190621 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:13.190647 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:13.190648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:13.190655 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:13.190701 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:13.191136 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.191606 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.192423 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.192669 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.192756 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:13.195014 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:13.195015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:13.195015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:13.195021 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:13.196560 (+ 1539us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:13.196923 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:13.196924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:13.196925 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:13.196982 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:13.197137 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:13.197137 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:13.197137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:13.197220 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:13.197326 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:13.197412 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:18:13.197477 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 00:18:13.199213 6442 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:13.198659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.0.98.1:35657)
0620 00:18:13.198697 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:13.198699 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:13.198762 (+ 63us) server_negotiation.cc:407] Connection header received
0620 00:18:13.198797 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:13.198797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:13.198805 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:13.198852 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:13.199162 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40090: BlockingRecv error: recv got EOF from 127.0.0.1:40090 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 00:18:13.199266 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1781914666918284-392-0/krb5kdc/krb5cc' of type FILE
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:13.200196 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:13.201021 392 init.cc:295] Successfully reacquired a new kerberos TGT
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[5931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914693, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:18:13.207811 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:13.201495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40102 (local address 127.0.98.1:35657)
0620 00:18:13.201565 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:13.201567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:13.201617 (+ 50us) server_negotiation.cc:407] Connection header received
0620 00:18:13.201653 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:13.201654 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:13.201662 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:13.201713 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:13.202034 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.202492 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.203165 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:13.203370 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:13.203425 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:13.205481 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:13.205483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:13.205483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:13.205489 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:13.206697 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:13.207046 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:13.207047 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:13.207047 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:13.207104 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:13.207309 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:13.207310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:13.207310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:13.207416 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:13.207518 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:13.207672 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 00:18:13.207748 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 00:18:13.208654 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 6038
I20260620 00:18:13.213878 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 6172
I20260620 00:18:13.218573 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 6306
I20260620 00:18:13.223047 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 5944
2026-06-20T00:18:13Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1899 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[6455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[6455](info): set up 2 sockets
Jun 20 00:18:13 dist-test-slave-r81b krb5kdc[6455](info): commencing operation
krb5kdc: starting...
W20260620 00:18:14.276444 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:14 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:14Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 00:18:14.399716 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34867
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34867
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:14.481618 6468 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:14.481902 6468 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:14.481940 6468 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:14.483168 6468 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:14.483243 6468 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:14.483268 6468 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:14.483289 6468 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:14.483315 6468 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:14.485016 6468 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6468
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:14.485359 6468 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:14.485620 6468 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:14.490972 6473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:14.490947 6476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:14.491010 6468 server_base.cc:1034] running on GCE node
W20260620 00:18:14.490942 6474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:14.491609 6468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:14.491931 6468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:14.493094 6468 hybrid_clock.cc:648] HybridClock initialized: now 1781914694493077 us; error 41 us; skew 500 ppm
Jun 20 00:18:14 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:14.498414 6468 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 00:18:14.498931 6468 webserver.cc:460] Webserver started at http://127.0.98.62:33125/ using document root <none> and password file <none>
I20260620 00:18:14.499185 6468 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:14.499255 6468 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:14.499402 6468 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:14.500607 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "b84405cafd7744b7b86c604bc3aa29a3"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "800be6bd18b47543bdf26098efb249fd"
server_key_iv: "fd53b7a213bc411ddc58af670de70d70"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.500996 6468 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "b84405cafd7744b7b86c604bc3aa29a3"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "800be6bd18b47543bdf26098efb249fd"
server_key_iv: "fd53b7a213bc411ddc58af670de70d70"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.502368 6468 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:14.503170 6483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:14.503378 6468 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:14.503485 6468 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "b84405cafd7744b7b86c604bc3aa29a3"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "800be6bd18b47543bdf26098efb249fd"
server_key_iv: "fd53b7a213bc411ddc58af670de70d70"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.503561 6468 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:14.513484 6468 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:14.515854 6468 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:14.516062 6468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:14.520339 6468 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34867
I20260620 00:18:14.520394 6535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34867 every 8 connection(s)
I20260620 00:18:14.520783 6468 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:14.521781 6536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:14.524113 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: Bootstrap starting.
I20260620 00:18:14.524674 6536 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:14.524784 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 6468
I20260620 00:18:14.524911 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:14.525024 6536 log.cc:824] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:14.525203 392 external_mini_cluster.cc:1428] Setting key aa21cc97329e5f6997d84ab2c59863d7
I20260620 00:18:14.525974 6536 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: No bootstrap required, opened a new log
I20260620 00:18:14.528119 6536 raft_consensus.cc:348] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } }
I20260620 00:18:14.528342 6536 raft_consensus.cc:374] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:14.528367 6536 raft_consensus.cc:729] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b84405cafd7744b7b86c604bc3aa29a3, State: Initialized, Role: FOLLOWER
I20260620 00:18:14.528513 6536 consensus_queue.cc:260] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } }
I20260620 00:18:14.528589 6536 raft_consensus.cc:388] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:14.528612 6536 raft_consensus.cc:482] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:14.528677 6536 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:14.529235 6536 raft_consensus.cc:504] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } }
I20260620 00:18:14.529331 6536 leader_election.cc:302] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b84405cafd7744b7b86c604bc3aa29a3; no voters:
I20260620 00:18:14.529564 6536 leader_election.cc:288] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:14.529588 6541 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:14.529692 6541 raft_consensus.cc:686] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 1 LEADER]: Becoming Leader. State: Replica: b84405cafd7744b7b86c604bc3aa29a3, State: Running, Role: LEADER
I20260620 00:18:14.529788 6541 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } }
I20260620 00:18:14.529857 6536 sys_catalog.cc:564] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:14.530114 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } }
I20260620 00:18:14.530182 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:14.530313 6541 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84405cafd7744b7b86c604bc3aa29a3. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } }
I20260620 00:18:14.530376 6541 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:14.530627 6545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:14.531131 6545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:18:14 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:14.536423 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:14.525811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40714 (local address 127.0.98.62:34867)
0620 00:18:14.525990 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:14.525998 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:14.526049 (+ 51us) server_negotiation.cc:407] Connection header received
0620 00:18:14.526501 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:14.526508 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:14.526557 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:14.526711 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:14.527323 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.528286 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.529203 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.529883 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.529985 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:14.533061 (+ 3076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:14.533071 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:14.533071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:14.533086 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:14.534821 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:14.535228 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:14.535231 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:14.535231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:14.535310 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:14.535513 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:14.535514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:14.535514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:14.535754 (+ 240us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:14.535875 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:14.536011 (+ 136us) server_negotiation.cc:299] Negotiation successful
0620 00:18:14.536114 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
I20260620 00:18:14.537642 6545 catalog_manager.cc:1347] Generated new cluster ID: ffe755da79b54db7922d4b1a25f4001d
I20260620 00:18:14.537722 6545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:14.541289 6545 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:14.541801 6545 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:14.545995 6545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: Generated new TSK 0
I20260620 00:18:14.546203 6545 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 00:18:14.571262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:14.654161 6562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:14.654412 6562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:14.654451 6562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:14.655659 6562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:14.655730 6562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:14.655778 6562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:14.657294 6562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:14.657560 6562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:14.657811 6562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:14.662885 6568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:14.662870 6567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:14.663132 6562 server_base.cc:1034] running on GCE node
W20260620 00:18:14.662874 6570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:14.663391 6562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:14.663677 6562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:14.664850 6562 hybrid_clock.cc:648] HybridClock initialized: now 1781914694664838 us; error 28 us; skew 500 ppm
Jun 20 00:18:14 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:14.667347 6562 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:18:14.667804 6562 webserver.cc:460] Webserver started at http://127.0.98.1:45915/ using document root <none> and password file <none>
I20260620 00:18:14.668056 6562 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:14.668123 6562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:14.668285 6562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:14.669368 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "fe5e2f41487944fea605bbd3e80a0e43"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "0cec679ab530e3ff426e94c5060c638f"
server_key_iv: "d42f85b9abbe3a9ccd0c01d516b49c67"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.669734 6562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fe5e2f41487944fea605bbd3e80a0e43"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "0cec679ab530e3ff426e94c5060c638f"
server_key_iv: "d42f85b9abbe3a9ccd0c01d516b49c67"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.671039 6562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:14.671739 6577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:14.671941 6562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:14.672039 6562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "fe5e2f41487944fea605bbd3e80a0e43"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "0cec679ab530e3ff426e94c5060c638f"
server_key_iv: "d42f85b9abbe3a9ccd0c01d516b49c67"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.672122 6562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:14.697305 6562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:14.699754 6562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:14.699929 6562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:14.700240 6562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:14.700575 6562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:14.700628 6562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:14.700664 6562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:14.700692 6562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:14.706638 6562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45141
I20260620 00:18:14.706745 6690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45141 every 8 connection(s)
I20260620 00:18:14.707085 6562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:18:14 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:14.716054 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:14.707858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34411 (local address 127.0.98.62:34867)
0620 00:18:14.707941 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:14.707943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:14.708096 (+ 153us) server_negotiation.cc:407] Connection header received
0620 00:18:14.708649 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:14.708650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:14.708660 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:14.708713 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:14.709619 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.710132 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.711304 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.711507 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.711565 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:14.713693 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:14.713694 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:14.713695 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:14.713701 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:14.714921 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:14.715278 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:14.715279 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:14.715279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:14.715327 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:14.715526 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:14.715526 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:14.715527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:14.715626 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:14.715675 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:14.715925 (+ 250us) server_negotiation.cc:299] Negotiation successful
0620 00:18:14.715987 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 00:18:14.716300 6693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:14.707812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.1:34411)
0620 00:18:14.707976 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:14.708012 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:14.708557 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:14.708770 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:14.708773 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:14.709344 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:14.709542 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:14.709546 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.710213 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.710213 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:14.711212 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:14.711214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.711583 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.711584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:14.711682 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:14.712269 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:14.712278 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:14.713610 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:14.714994 (+ 1384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:14.714999 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:14.715001 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:14.715214 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:14.715380 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:14.715381 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:14.715381 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:14.715474 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:14.715678 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:14.715682 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:14.715814 (+ 132us) client_negotiation.cc:769] Sending connection context
0620 00:18:14.715898 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:18:14.716017 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260620 00:18:14.716719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 6562
I20260620 00:18:14.716866 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:14.716952 6691 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34867
I20260620 00:18:14.717058 6691 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:14.717097 392 external_mini_cluster.cc:1428] Setting key 26c64db09f1ac9d56844beef2c2649a5
I20260620 00:18:14.717259 6691 heartbeater.cc:511] Master 127.0.98.62:34867 requested a full tablet report, sending...
I20260620 00:18:14.717809 6499 ts_manager.cc:194] Registered new tserver with Master: fe5e2f41487944fea605bbd3e80a0e43 (127.0.98.1:45141)
I20260620 00:18:14.718367 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:34411
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 00:18:14.743290 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:14.825075 6696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:14.825337 6696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:14.825381 6696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:14.826509 6696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:14.826577 6696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:14.826622 6696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:14.828202 6696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6696
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:14.828475 6696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:14.828713 6696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:18:14.834208 6696 server_base.cc:1034] running on GCE node
W20260620 00:18:14.834174 6702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:14.834175 6704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:14.834174 6701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:14.834614 6696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:14.834828 6696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:14.836001 6696 hybrid_clock.cc:648] HybridClock initialized: now 1781914694835995 us; error 34 us; skew 500 ppm
Jun 20 00:18:14 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:14.838768 6696 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:18:14.839211 6696 webserver.cc:460] Webserver started at http://127.0.98.2:36323/ using document root <none> and password file <none>
I20260620 00:18:14.839468 6696 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:14.839540 6696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:14.839685 6696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:14.840831 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "c78bd803c43a4eee831fdba42b624364"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "162ed18ebf042a1199c14c369c82baad"
server_key_iv: "8edf0f70bf1ae3a9a59ef64b36c8d7b0"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.841190 6696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c78bd803c43a4eee831fdba42b624364"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "162ed18ebf042a1199c14c369c82baad"
server_key_iv: "8edf0f70bf1ae3a9a59ef64b36c8d7b0"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.842540 6696 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:14.843293 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:14.843453 6696 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:14.843560 6696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "c78bd803c43a4eee831fdba42b624364"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "162ed18ebf042a1199c14c369c82baad"
server_key_iv: "8edf0f70bf1ae3a9a59ef64b36c8d7b0"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.843633 6696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:14.856560 6696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:14.858923 6696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:14.859087 6696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:14.859334 6696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:14.859644 6696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:14.859697 6696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:14.859746 6696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:14.859771 6696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:14.865674 6696 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40151
I20260620 00:18:14.865777 6824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40151 every 8 connection(s)
I20260620 00:18:14.866087 6696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:14.868621 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 6696
I20260620 00:18:14.868726 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:14.868963 392 external_mini_cluster.cc:1428] Setting key 3c04fba4952e003bb3eb661cb6a89087
Jun 20 00:18:14 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:18:14.877279 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:14.866816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47957 (local address 127.0.98.62:34867)
0620 00:18:14.866911 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:14.866913 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:14.867192 (+ 279us) server_negotiation.cc:407] Connection header received
0620 00:18:14.867790 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:14.867792 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:14.867806 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:14.867871 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:14.869133 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.869774 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.870835 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.871065 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.871164 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:14.873926 (+ 2762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:14.873927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:14.873928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:14.873937 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:14.875656 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:14.876167 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:14.876169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:14.876169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:14.876236 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:14.876552 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:14.876553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:14.876553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:14.876691 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:14.876779 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:14.877084 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 00:18:14.877177 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 00:18:14.877513 6827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:14.866822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.2:47957)
0620 00:18:14.867042 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:14.867089 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:14.867669 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:14.867951 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:14.867954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:14.868646 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:14.868952 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:14.868958 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.869858 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.869860 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:14.870723 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:14.870726 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:14.871176 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:14.871177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:14.871325 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:14.871972 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:14.871981 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:14.873792 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:14.875770 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:14.875779 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:14.875781 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:14.876038 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:14.876350 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:14.876351 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:14.876351 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:14.876475 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:14.876768 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:14.876775 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:14.876937 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 00:18:14.877055 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 00:18:14.877234 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260620 00:18:14.878089 6825 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34867
I20260620 00:18:14.878219 6825 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:14.878419 6825 heartbeater.cc:511] Master 127.0.98.62:34867 requested a full tablet report, sending...
I20260620 00:18:14.878851 6499 ts_manager.cc:194] Registered new tserver with Master: c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151)
I20260620 00:18:14.879245 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:47957
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 00:18:14.897181 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:14.981534 6830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:14.981779 6830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:14.981818 6830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:14.983013 6830 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:14.983083 6830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:14.983130 6830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:14.984701 6830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34867
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6830
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:14.985018 6830 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:14.985252 6830 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:14.990831 6836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:14.990834 6835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:14.990834 6838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:14.991074 6830 server_base.cc:1034] running on GCE node
I20260620 00:18:14.991285 6830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:14.991485 6830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:14.992633 6830 hybrid_clock.cc:648] HybridClock initialized: now 1781914694992619 us; error 26 us; skew 500 ppm
Jun 20 00:18:14 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:14.995337 6830 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:18:14.995728 6830 webserver.cc:460] Webserver started at http://127.0.98.3:38199/ using document root <none> and password file <none>
I20260620 00:18:14.995926 6830 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:14.995968 6830 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:14.996054 6830 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:14.997159 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "47f818f1a0f14277b5fb1a1a88ce472b"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "a88c629fb8a1258af7baeb53806e8687"
server_key_iv: "acda027f63978d7bb651a67e3b437529"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.997464 6830 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47f818f1a0f14277b5fb1a1a88ce472b"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "a88c629fb8a1258af7baeb53806e8687"
server_key_iv: "acda027f63978d7bb651a67e3b437529"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.998705 6830 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:18:14.999424 6845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:14.999589 6830 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:14.999699 6830 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "47f818f1a0f14277b5fb1a1a88ce472b"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "a88c629fb8a1258af7baeb53806e8687"
server_key_iv: "acda027f63978d7bb651a67e3b437529"
server_key_version: "encryptionkey@0"
I20260620 00:18:14.999781 6830 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:15.029736 6830 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:15.032364 6830 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:15.032532 6830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:15.032773 6830 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:15.033115 6830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:15.033172 6830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.033218 6830 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:15.033246 6830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.039148 6830 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38931
I20260620 00:18:15.039187 6958 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38931 every 8 connection(s)
I20260620 00:18:15.039597 6830 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:15.043927 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 6830
I20260620 00:18:15.044026 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:15.044226 392 external_mini_cluster.cc:1428] Setting key 82a648b5928b0fa0dd90c179aa44acad
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:15.049369 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.040384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41719 (local address 127.0.98.62:34867)
0620 00:18:15.040462 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.040465 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.040651 (+ 186us) server_negotiation.cc:407] Connection header received
0620 00:18:15.041142 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.041144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.041154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.041212 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.042186 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.042676 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.043652 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.043856 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.043912 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.046501 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.046502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.046503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.046511 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.047992 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.048451 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.048452 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.048452 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.048503 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.048796 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.048797 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.048797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.048903 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.048956 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.049221 (+ 265us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.049303 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:18:15.049562 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.040351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.3:41719)
0620 00:18:15.040529 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.040565 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.041026 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.041269 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.041273 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.041886 (+ 613us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.042083 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.042088 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.042757 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.042758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.043577 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.043579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.043946 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.043947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.044109 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.044754 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.044765 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.046407 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.048066 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.048071 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.048073 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.048387 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.048596 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.048597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.048597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.048725 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.048974 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.048979 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.049106 (+ 127us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.049197 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.049345 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":105,"threads_started":1}
I20260620 00:18:15.050076 6959 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34867
I20260620 00:18:15.050170 6959 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:15.050329 6959 heartbeater.cc:511] Master 127.0.98.62:34867 requested a full tablet report, sending...
I20260620 00:18:15.050733 6499 ts_manager.cc:194] Registered new tserver with Master: 47f818f1a0f14277b5fb1a1a88ce472b (127.0.98.3:38931)
I20260620 00:18:15.051082 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41719
I20260620 00:18:15.055895 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260620 00:18:15.082780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36989
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34867,127.0.98.61:36989
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:15.167447 6964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:15.167663 6964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:15.167686 6964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:15.168982 6964 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:15.169029 6964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:15.169044 6964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:15.169056 6964 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:15.169095 6964 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:15.170581 6964 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34867,127.0.98.61:36989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6964
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
W20260620 00:18:15.170708 6964 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:34867,127.0.98.61:36989'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260620 00:18:15.170863 6964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:15.171105 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:15.176220 6969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:15.176236 6972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.176348 6964 server_base.cc:1034] running on GCE node
W20260620 00:18:15.176242 6970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.176728 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:15.177004 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:15.178141 6964 hybrid_clock.cc:648] HybridClock initialized: now 1781914695178117 us; error 36 us; skew 500 ppm
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:15.180596 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 00:18:15.181051 6964 webserver.cc:460] Webserver started at http://127.0.98.61:43407/ using document root <none> and password file <none>
I20260620 00:18:15.181242 6964 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:15.181290 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:15.181380 6964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:15.182377 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data/instance:
uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "867a099a7414d08c96c10679e55848e5"
server_key_iv: "16dae5bc444ab9f3c6f356c15568cf02"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.182677 6964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal/instance:
uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "867a099a7414d08c96c10679e55848e5"
server_key_iv: "16dae5bc444ab9f3c6f356c15568cf02"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.183984 6964 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:18:15.184741 6979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.184911 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.185011 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal
uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "867a099a7414d08c96c10679e55848e5"
server_key_iv: "16dae5bc444ab9f3c6f356c15568cf02"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.185084 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:15.193554 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:15.195288 6964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:15.195439 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:15.199522 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36989
I20260620 00:18:15.199559 7031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36989 every 8 connection(s)
I20260620 00:18:15.199914 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:18:15.200961 7032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:15.202461 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:18:15.208048 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 6964
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:15.212213 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.203230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40730 (local address 127.0.98.62:34867)
0620 00:18:15.203315 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.203317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.203498 (+ 181us) server_negotiation.cc:407] Connection header received
0620 00:18:15.204035 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.204036 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.204046 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.204105 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.204799 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.205295 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.206487 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.206770 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.206887 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.209370 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.209372 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.209372 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.209387 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.210720 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.211143 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.211144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.211145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.211202 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.211472 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.211473 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.211473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.211626 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.211707 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.211996 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.212080 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 00:18:15.212393 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.203191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:40730)
0620 00:18:15.203374 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.203415 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.203931 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.204206 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.204210 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.204465 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.204669 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.204675 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.205397 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.205398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.206355 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.206357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.206883 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.206884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.206994 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.207579 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.207588 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.209269 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.210816 (+ 1547us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.210822 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.210825 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.211052 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.211288 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.211289 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.211289 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.211404 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.211697 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.211703 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.211862 (+ 159us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.211977 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.212132 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":60,"threads_started":1}
I20260620 00:18:15.213021 7032 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:18:15.217994 7036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.208651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43856 (local address 127.0.98.61:36989)
0620 00:18:15.208805 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.208810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.208854 (+ 44us) server_negotiation.cc:407] Connection header received
0620 00:18:15.208923 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.208926 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.208948 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.209028 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.209841 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.210673 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.211431 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.211764 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.211862 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.214837 (+ 2975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.214848 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.214849 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.214856 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.216682 (+ 1826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.217105 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.217109 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.217109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.217199 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.217419 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.217420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.217421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.217569 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.217760 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.217803 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.217878 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260620 00:18:15.220757 7034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.213275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.0.1:43866)
0620 00:18:15.213321 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.213332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.213370 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.213619 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.213620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.213759 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.213811 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.213813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.214478 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.214479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.215109 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.215112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.215765 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.215766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.215842 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.216182 (+ 340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.216187 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.217361 (+ 1174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.219723 (+ 2362us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.219724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.219724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.219885 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.220088 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.220089 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.220089 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.220170 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.220424 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.220425 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.220461 (+ 36us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.220527 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.220623 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260620 00:18:15.220794 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.213321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43866 (local address 127.0.98.61:36989)
0620 00:18:15.213487 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.213489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.213496 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:18:15.213517 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.213517 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.213529 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.213609 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.213890 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.214405 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.215204 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.215656 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.215754 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.217458 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.217459 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.217459 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.217467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.219621 (+ 2154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.219967 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.219968 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.219969 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.220025 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.220257 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.220258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.220258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.220360 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.220433 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.220554 (+ 121us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.220672 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":56,"server-negotiator.queue_time_us":145,"thread_start_us":101,"threads_started":1}
I20260620 00:18:15.221952 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Bootstrap starting.
I20260620 00:18:15.222569 7032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:15.222849 7032 log.cc:824] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:15.223666 7032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: No bootstrap required, opened a new log
I20260620 00:18:15.225097 7032 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } }
I20260620 00:18:15.225224 7032 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:15.225284 7032 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be6c69cb34f46079b9c5c5c45ad8ed1, State: Initialized, Role: FOLLOWER
I20260620 00:18:15.225492 7032 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } }
I20260620 00:18:15.225595 7039 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:15.225718 7039 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } }
I20260620 00:18:15.225783 7032 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:15.225943 7040 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } } }
I20260620 00:18:15.226013 7039 leader_election.cc:288] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b84405cafd7744b7b86c604bc3aa29a3 (127.0.98.62:34867)
I20260620 00:18:15.226017 7040 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:18:15.226393 6510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b84405cafd7744b7b86c604bc3aa29a3" is_pre_election: true
I20260620 00:18:15.226833 6981 leader_election.cc:302] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 2be6c69cb34f46079b9c5c5c45ad8ed1; no voters: b84405cafd7744b7b86c604bc3aa29a3
I20260620 00:18:15.226960 7040 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260620 00:18:15.227970 7051 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:18:15.228039 7051 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:18:15.229166 6964 master_runner.cc:428] Detected that this master 2be6c69cb34f46079b9c5c5c45ad8ed1 is joining an existing cluster
I20260620 00:18:15.229214 6964 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:36989
I20260620 00:18:15.237126 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.230232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.0.1:43876)
0620 00:18:15.230541 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.230551 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.230609 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.230731 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.230731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.230944 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.231014 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.231016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.231955 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.231956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.232588 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.232590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.233080 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.233081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.233133 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.233659 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.233664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.233983 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.235857 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.235858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.235858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.236033 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.236338 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.236340 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.236340 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.236426 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.236774 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.236775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.236820 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.236894 (+ 74us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.237008 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":72,"threads_started":1}
I20260620 00:18:15.237367 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.230405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.0.98.61:36989)
0620 00:18:15.230440 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.230442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.230620 (+ 178us) server_negotiation.cc:407] Connection header received
0620 00:18:15.230646 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.230647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.230659 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.230727 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.231082 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.231856 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.232665 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.232985 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.233065 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.234068 (+ 1003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.234070 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.234070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.234077 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.235737 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.236111 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.236112 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.236113 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.236238 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.236503 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.236504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.236504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.236690 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.236772 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.236932 (+ 160us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.237084 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":60,"server-negotiator.queue_time_us":17}
I20260620 00:18:15.243844 7058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.230258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:40740)
0620 00:18:15.237276 (+ 7018us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.237291 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.237401 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.237648 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.237649 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.237844 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.237903 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.237905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.238859 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.238861 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.239479 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.239481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.240011 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.240011 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.240070 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.240736 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.240741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.241013 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.242827 (+ 1814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.242829 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.242829 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.243015 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.243262 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.243263 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.243263 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.243325 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.243594 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.243595 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.243632 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.243683 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.243772 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7004,"thread_start_us":96,"threads_started":1}
I20260620 00:18:15.243903 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.230304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40740 (local address 127.0.98.62:34867)
0620 00:18:15.230355 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.230357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.237361 (+ 7004us) server_negotiation.cc:407] Connection header received
0620 00:18:15.237505 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.237508 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.237525 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.237574 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.237976 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.238732 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.239578 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.239916 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.239987 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.241110 (+ 1123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.241111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.241111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.241120 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.242708 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.243087 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.243088 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.243088 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.243149 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.243390 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.243391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.243391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.243531 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.243599 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.243723 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.243806 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:18:15.254127 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.246413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.0.1:43892)
0620 00:18:15.246571 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.246618 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.246691 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.246840 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.246844 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.247109 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.247159 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.247161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.248038 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.248039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.248698 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.248701 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.249160 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.249161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.249222 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.249710 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.249715 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.251260 (+ 1545us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.253066 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.253073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.253075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.253310 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.253509 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.253510 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.253510 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.253565 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.253824 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.253829 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.253874 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.253939 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.254037 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":96,"threads_started":1}
I20260620 00:18:15.254156 7037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.246485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43892 (local address 127.0.98.61:36989)
0620 00:18:15.246525 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.246527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.246688 (+ 161us) server_negotiation.cc:407] Connection header received
0620 00:18:15.246757 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.246757 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.246767 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.246859 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.247228 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.247958 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.248785 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.249084 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.249157 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.251347 (+ 2190us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.251349 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.251349 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.251357 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.252979 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.253385 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.253386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.253387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.253445 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.253623 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.253623 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.253624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.253762 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.253830 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.253984 (+ 154us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.254062 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:18:15.255016 6499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:36989: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 36989
}
attrs {
promote: true
}
}
dest_uuid: "b84405cafd7744b7b86c604bc3aa29a3"
cas_config_opid_index: -1
I20260620 00:18:15.255414 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } }
I20260620 00:18:15.255934 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 2be6c69cb34f46079b9c5c5c45ad8ed1 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } } }
I20260620 00:18:15.256280 7006 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:15.256958 7061 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:36989
I20260620 00:18:15.257171 6964 master.cc:537] Master@127.0.98.61:36989 shutting down...
I20260620 00:18:15.257207 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } } }
I20260620 00:18:15.257283 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:15.257611 7006 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Refusing update from remote peer b84405cafd7744b7b86c604bc3aa29a3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260620 00:18:15.257994 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:18:15.258700 6484 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260620 00:18:15.259267 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 -> Peer 2be6c69cb34f46079b9c5c5c45ad8ed1 (127.0.98.61:36989): Couldn't send request to peer 2be6c69cb34f46079b9c5c5c45ad8ed1. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 00:18:15.260965 6964 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:15.261050 6964 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:15.261094 6964 tablet_replica.cc:330] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: stopping tablet replica
I20260620 00:18:15.274279 6964 master.cc:552] Master@127.0.98.61:36989 shutdown complete.
I20260620 00:18:15.274360 6964 master_runner.cc:305] Clearing existing system tablet
I20260620 00:18:15.274503 6964 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:15.274570 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:15.275316 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:15.275816 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.275976 6964 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:15.276033 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.276055 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal
uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "867a099a7414d08c96c10679e55848e5"
server_key_iv: "16dae5bc444ab9f3c6f356c15568cf02"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.276764 6964 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:15.278097 6964 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 00:18:15.278154 6964 log.cc:1196] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260620 00:18:15.278350 6964 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Deleting consensus metadata
I20260620 00:18:15.278484 6964 master_runner.cc:315] Copying system tablet from 127.0.98.62:34867
I20260620 00:18:15.279145 6964 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34867
I20260620 00:18:15.285617 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.279547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.0.98.62:34867)
0620 00:18:15.279594 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.279597 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.279660 (+ 63us) server_negotiation.cc:407] Connection header received
0620 00:18:15.279741 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.279742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.279751 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.279799 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.280278 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.280777 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.281851 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.282033 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.282089 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.283136 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.283137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.283137 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.283143 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.284607 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.284970 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.284970 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.284971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.285018 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.285210 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.285210 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.285210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.285306 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.285354 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.285499 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.285554 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:18:15.285818 7071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.279459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:40746)
0620 00:18:15.279583 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.279595 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.279672 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.279807 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.279807 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.280019 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.280086 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.280088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.280859 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.280860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.281766 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.281768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.282107 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.282108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.282182 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.282659 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.282664 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.283050 (+ 386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.284682 (+ 1632us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.284683 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.284683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.284903 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.285079 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.285080 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.285080 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.285150 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.285364 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.285365 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.285410 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.285485 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.285585 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260620 00:18:15.286212 6520 tablet_copy_service.cc:140] P b84405cafd7744b7b86c604bc3aa29a3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 2be6c69cb34f46079b9c5c5c45ad8ed1 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40746)
I20260620 00:18:15.286303 6520 tablet_copy_service.cc:161] P b84405cafd7744b7b86c604bc3aa29a3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 2be6c69cb34f46079b9c5c5c45ad8ed1 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40746: session id = 2be6c69cb34f46079b9c5c5c45ad8ed1-00000000000000000000000000000000
I20260620 00:18:15.287148 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 00:18:15.287781 6964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 00:18:15.288797 6964 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: tablet copy: Starting download of 0 data blocks...
I20260620 00:18:15.288961 6964 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: tablet copy: Starting download of 1 WAL segments...
I20260620 00:18:15.289794 6964 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 00:18:15.290674 6520 tablet_copy_service.cc:342] P b84405cafd7744b7b86c604bc3aa29a3: Request end of tablet copy session 2be6c69cb34f46079b9c5c5c45ad8ed1-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:40746
I20260620 00:18:15.290735 6520 tablet_copy_service.cc:433] P b84405cafd7744b7b86c604bc3aa29a3: ending tablet copy session 2be6c69cb34f46079b9c5c5c45ad8ed1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 2be6c69cb34f46079b9c5c5c45ad8ed1
I20260620 00:18:15.293265 6964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:15.294612 7073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:15.294618 7072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.295200 6964 server_base.cc:1034] running on GCE node
W20260620 00:18:15.295282 7075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.295435 6964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:15.295579 6964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:15.296696 6964 hybrid_clock.cc:648] HybridClock initialized: now 1781914695296686 us; error 27 us; skew 500 ppm
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:15.298151 6964 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260620 00:18:15.298413 6964 webserver.cc:460] Webserver started at http://127.0.98.61:36733/ using document root <none> and password file <none>
I20260620 00:18:15.298523 6964 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:15.298558 6964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:15.299243 6964 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:15.299803 7082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.299947 6964 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:15.300015 6964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal
uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "867a099a7414d08c96c10679e55848e5"
server_key_iv: "16dae5bc444ab9f3c6f356c15568cf02"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.300076 6964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:15.326726 6964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:15.328558 6964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:15.331861 6964 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36989
I20260620 00:18:15.331897 7134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36989 every 8 connection(s)
I20260620 00:18:15.332243 6964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:18:15.332945 7135 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:18:15.333364 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Bootstrap starting.
I20260620 00:18:15.335592 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:18:15.335811 7135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Bootstrap complete.
I20260620 00:18:15.336016 7135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } }
I20260620 00:18:15.336102 7135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 2be6c69cb34f46079b9c5c5c45ad8ed1, State: Initialized, Role: LEARNER
I20260620 00:18:15.336143 7135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } }
I20260620 00:18:15.336351 7135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:15.336369 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } } }
I20260620 00:18:15.336468 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:18:15.337926 7149 catalog_manager.cc:1259] Loaded cluster ID: ffe755da79b54db7922d4b1a25f4001d
I20260620 00:18:15.337978 7149 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: loading cluster ID for follower catalog manager: success
I20260620 00:18:15.339097 7149 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: acquiring CA information for follower catalog manager: success
I20260620 00:18:15.339434 7149 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:18:15.378018 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.370309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43914 (local address 127.0.98.61:36989)
0620 00:18:15.370466 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.370468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.370478 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:15.370506 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.370507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.370518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.370615 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.371301 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.372419 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.373258 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.373578 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.373651 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.375004 (+ 1353us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.375006 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.375007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.375016 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.376700 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.377022 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.377023 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.377023 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.377086 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.377347 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.377348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.377348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.377493 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.377609 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.377800 (+ 191us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.377895 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260620 00:18:15.405483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44061
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34867,127.0.98.61:36989,127.0.98.60:44061
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:15.491339 7153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:15.491550 7153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:15.491573 7153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:15.492836 7153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:15.492879 7153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:15.492895 7153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:15.492937 7153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:15.492964 7153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:15.494467 7153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34867,127.0.98.61:36989,127.0.98.60:44061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:44061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:15.494717 7153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:15.494952 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:15.500119 7159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:15.500098 7161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.500270 7153 server_base.cc:1034] running on GCE node
W20260620 00:18:15.500372 7158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.500625 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:15.500914 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:15.502066 7153 hybrid_clock.cc:648] HybridClock initialized: now 1781914695502048 us; error 34 us; skew 500 ppm
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:15.504772 7153 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 00:18:15.505240 7153 webserver.cc:460] Webserver started at http://127.0.98.60:44347/ using document root <none> and password file <none>
I20260620 00:18:15.505438 7153 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:15.505479 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:15.505566 7153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:15.506683 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data/instance:
uuid: "04c7cfc64ea34fb388160a3e504c41cd"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "9f2f9814cf00042df7a715e91eed1bcd"
server_key_iv: "acd9641253d943c910007b0bb7da6d4e"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.507028 7153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal/instance:
uuid: "04c7cfc64ea34fb388160a3e504c41cd"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "9f2f9814cf00042df7a715e91eed1bcd"
server_key_iv: "acd9641253d943c910007b0bb7da6d4e"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.508430 7153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:18:15.509143 7168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.509491 7153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:15.509564 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal
uuid: "04c7cfc64ea34fb388160a3e504c41cd"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "9f2f9814cf00042df7a715e91eed1bcd"
server_key_iv: "acd9641253d943c910007b0bb7da6d4e"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.509658 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:15.540277 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:15.542282 7153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:15.542447 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:15.546624 7153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44061
I20260620 00:18:15.546656 7220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44061 every 8 connection(s)
I20260620 00:18:15.547035 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:18:15.548218 7221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:15.549824 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:18:15.550937 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 7153
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:18:15.560832 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.550813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40752 (local address 127.0.98.62:34867)
0620 00:18:15.550866 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.550869 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.551058 (+ 189us) server_negotiation.cc:407] Connection header received
0620 00:18:15.551719 (+ 661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.551720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.551734 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.551800 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.552671 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.553530 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.554806 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.555096 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.555168 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.557775 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.557777 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.557777 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.557783 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.559260 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.559735 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.559736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.559736 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.559789 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.560057 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.560057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.560057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.560246 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.560302 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.560640 (+ 338us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.560717 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:18:15.560952 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.550616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:40752)
0620 00:18:15.550851 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.550906 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.551598 (+ 692us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.551876 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.551880 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.552314 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.552566 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.552572 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.553630 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.553631 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.554713 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.554716 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.555178 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.555179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.555282 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.555854 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.555867 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.557682 (+ 1815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.559339 (+ 1657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.559346 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.559349 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.559651 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.559856 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.559857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.559857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.559988 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.560313 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.560318 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.560491 (+ 173us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.560603 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.560740 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":122,"threads_started":1}
I20260620 00:18:15.561543 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:18:15.561831 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.551682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39964 (local address 127.0.98.60:44061)
0620 00:18:15.551855 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.551859 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.551877 (+ 18us) server_negotiation.cc:407] Connection header received
0620 00:18:15.551907 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.551908 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.551926 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.551996 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.552762 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.553897 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.554903 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.555343 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.555394 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.558618 (+ 3224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.558626 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.558626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.558633 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.560491 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.560892 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.560895 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.560896 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.560951 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.561194 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.561195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.561195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.561367 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.561506 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.561610 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.561705 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":101,"threads_started":1}
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260620 00:18:15.567683 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.561811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.0.1:43924)
0620 00:18:15.561844 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.561855 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.561892 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.562276 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.562276 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.562424 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.562494 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.562496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.563095 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.563096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.563598 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.563600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.563927 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.563928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.563980 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.564424 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.564429 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.565504 (+ 1075us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.566805 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.566806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.566806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.566959 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.567161 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.567162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.567162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.567226 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.567455 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.567455 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.567492 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.567550 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.567619 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 00:18:15.567687 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.562064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43924 (local address 127.0.98.61:36989)
0620 00:18:15.562120 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.562123 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.562131 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:18:15.562157 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.562158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.562171 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.562229 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.562559 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.563034 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.563664 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.563865 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.563912 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.565576 (+ 1664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.565578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.565578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.565584 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.566712 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.567023 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.567023 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.567024 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.567071 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.567291 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.567292 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.567292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.567385 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.567434 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.567565 (+ 131us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.567628 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:18:15.568270 7221 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:18:15.574014 7223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.568513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.0.1:39978)
0620 00:18:15.568560 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.568571 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.568607 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.568703 (+ 96us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.568703 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.568813 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.568842 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.568843 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.569412 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.569412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.569881 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.569882 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.570188 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.570188 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.570234 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.570681 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.570686 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.571805 (+ 1119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.573149 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.573150 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.573150 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.573312 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.573479 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.573479 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.573479 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.573528 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.573799 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.573799 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.573833 (+ 34us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.573886 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.573950 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260620 00:18:15.574201 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.568561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.0.98.60:44061)
0620 00:18:15.568613 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.568614 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.568639 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:18:15.568654 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.568655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.568664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.568699 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.568897 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.569343 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.569945 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.570128 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.570176 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.571883 (+ 1707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.571884 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.571884 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.571890 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.573080 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.573374 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.573375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.573375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.573420 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.573603 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.573604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.573605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.573733 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.573799 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.573972 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.574049 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:18:15.575567 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: Bootstrap starting.
I20260620 00:18:15.576491 7221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:15.576774 7221 log.cc:824] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:15.577493 7221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: No bootstrap required, opened a new log
I20260620 00:18:15.579036 7221 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } }
I20260620 00:18:15.579221 7221 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:15.579270 7221 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04c7cfc64ea34fb388160a3e504c41cd, State: Initialized, Role: FOLLOWER
I20260620 00:18:15.579375 7221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } }
I20260620 00:18:15.579620 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } } }
I20260620 00:18:15.579695 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:18:15.579701 7221 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: configured and running, proceeding with master startup.
W20260620 00:18:15.581421 7237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:18:15.581482 7237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:18:15.583691 7153 master_runner.cc:428] Detected that this master 04c7cfc64ea34fb388160a3e504c41cd is joining an existing cluster
I20260620 00:18:15.583750 7153 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:44061
I20260620 00:18:15.592295 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.584911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40760 (local address 127.0.98.62:34867)
0620 00:18:15.584957 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.584959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.585095 (+ 136us) server_negotiation.cc:407] Connection header received
0620 00:18:15.585156 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.585157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.585167 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.585255 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.585690 (+ 435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.586504 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.587572 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.587883 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.587942 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.589034 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.589036 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.589037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.589044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.590748 (+ 1704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.591400 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.591402 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.591402 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.591462 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.591700 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.591701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.591702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.591866 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.591925 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.592089 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.592193 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:18:15.592315 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.584874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:40760)
0620 00:18:15.585016 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.585027 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.585091 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.585262 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.585262 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.585552 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.585628 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.585630 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.586587 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.586587 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.587486 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.587488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.587995 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.587996 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.588059 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.588596 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.588599 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.588935 (+ 336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.590834 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.590835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.590836 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.591320 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.591539 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.591540 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.591541 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.591621 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.591930 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.591931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.591976 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.592064 (+ 88us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.592200 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260620 00:18:15.593529 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.584958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43934 (local address 127.0.98.61:36989)
0620 00:18:15.585001 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.585003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.585454 (+ 451us) server_negotiation.cc:407] Connection header received
0620 00:18:15.585507 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.585508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.585518 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.585577 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.585959 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.586703 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.588744 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.589044 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.589085 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.590213 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.590214 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.590214 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.590222 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.591757 (+ 1535us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.592181 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.592183 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.592183 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.592245 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.592934 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.592936 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.592936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.593073 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.593127 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.593345 (+ 218us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.593434 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:18:15.594090 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.585047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.0.1:43934)
0620 00:18:15.585377 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.585387 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.585448 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.585581 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.585582 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.585798 (+ 216us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.585862 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.585864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.586805 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.586805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.588634 (+ 1829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.588636 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.589172 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.589173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.589237 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.589734 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.589739 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.590120 (+ 381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.591842 (+ 1722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.591843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.591843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.592055 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.592319 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.592320 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.592320 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.592847 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.593178 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.593179 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.593232 (+ 53us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.593870 (+ 638us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.593986 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":67,"threads_started":1}
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914694, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:18:15.611797 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.594631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39994 (local address 127.0.98.60:44061)
0620 00:18:15.594804 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.594806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.594813 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:18:15.594851 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.594852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.594860 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.594913 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.595257 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.595717 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.596769 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.596969 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.597021 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.598893 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.598894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.598895 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.598902 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.610385 (+ 11483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.610839 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.610840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.610841 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.610903 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.611135 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.611136 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.611137 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.611303 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.611392 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.611571 (+ 179us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.611669 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":128,"threads_started":1}
I20260620 00:18:15.611795 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.594587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.0.1:39994)
0620 00:18:15.594633 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.594652 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.594687 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.594942 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.594943 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.595107 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.595155 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.595157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.595797 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.595798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.596693 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.596696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.597040 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.597041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.597111 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.597500 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.597504 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.598807 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.610484 (+ 11677us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.610486 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.610486 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.610751 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.610995 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.610997 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.610997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.611064 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.611389 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.611390 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.611450 (+ 60us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.611542 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.611669 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 00:18:15.612309 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.585317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39988 (local address 127.0.98.60:44061)
0620 00:18:15.585353 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.585354 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.585783 (+ 429us) server_negotiation.cc:407] Connection header received
0620 00:18:15.585855 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.585856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.585862 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.585886 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.586233 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.586892 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.587836 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.588115 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.588205 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.609504 (+ 21299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.609506 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.609506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.609513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.610856 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.611219 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.611220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.611220 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.611264 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.611487 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.611488 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.611489 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.611608 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.611666 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.612116 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.612210 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":14,"server-negotiator.queue_time_us":19}
I20260620 00:18:15.612473 6499 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:44061: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 44061
}
attrs {
promote: true
}
}
dest_uuid: "b84405cafd7744b7b86c604bc3aa29a3"
cas_config_opid_index: 5
I20260620 00:18:15.612623 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.585255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.0.1:39988)
0620 00:18:15.585737 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.585746 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.585801 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.585930 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.585930 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.586110 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.586163 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.586165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.586970 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.586970 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.587762 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.587763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.588271 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.588271 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.588355 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.608778 (+ 20423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.608783 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.609396 (+ 613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.610938 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.610939 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.610939 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.611149 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.611332 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.611333 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.611333 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.611396 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.611688 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.611689 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.611757 (+ 68us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.612421 (+ 664us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.612533 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":63,"threads_started":1}
I20260620 00:18:15.612771 6499 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } }
I20260620 00:18:15.613137 7060 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 04c7cfc64ea34fb388160a3e504c41cd (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.614130 7060 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:44061
I20260620 00:18:15.614214 7060 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.614284 7060 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:15.614401 7153 master.cc:537] Master@127.0.98.60:44061 shutting down...
W20260620 00:18:15.614997 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 -> Peer 04c7cfc64ea34fb388160a3e504c41cd (127.0.98.60:44061): Couldn't send request to peer 04c7cfc64ea34fb388160a3e504c41cd. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260620 00:18:15.618180 7153 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 0 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:15.618286 7153 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 0 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:15.618338 7153 tablet_replica.cc:330] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: stopping tablet replica
I20260620 00:18:15.619884 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.614567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.0.1:43936)
0620 00:18:15.615090 (+ 523us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.615100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.615140 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.616002 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.616003 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.616006 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:15.616049 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.616050 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.617443 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.617444 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.618681 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.618684 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.619572 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.619573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.619650 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.619683 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.619736 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.619785 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":506}
W20260620 00:18:15.620118 6453 connection.cc:535] client connection to 127.0.98.60:44061 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:18:15.620394 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.614851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43936 (local address 127.0.98.61:36989)
0620 00:18:15.615844 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.615846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.615856 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:15.615879 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.615879 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.615903 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.615973 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:15.616188 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.617346 (+ 1158us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.618757 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.619384 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.619449 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.619537 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.620269 (+ 732us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.620318 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":968}
I20260620 00:18:15.620704 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 LEARNER]: Refusing update from remote peer b84405cafd7744b7b86c604bc3aa29a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260620 00:18:15.620936 7247 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260620 00:18:15.621497 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 04c7cfc64ea34fb388160a3e504c41cd (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.622392 7249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.622484 7249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:18:15.622627 7247 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: attempting to promote NON_VOTER 2be6c69cb34f46079b9c5c5c45ad8ed1 to VOTER
I20260620 00:18:15.622642 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84405cafd7744b7b86c604bc3aa29a3. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: true } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.622685 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: This master's current role is: LEARNER
I20260620 00:18:15.622849 7247 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } }
I20260620 00:18:15.623391 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.623303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.0.1:40000)
0620 00:18:15.623332 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.623369 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
I20260620 00:18:15.623471 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 LEARNER]: Refusing update from remote peer b84405cafd7744b7b86c604bc3aa29a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260620 00:18:15.623706 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.623623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.0.1:40004)
0620 00:18:15.623656 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.623681 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44061: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 00:18:15.623723 7060 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260620 00:18:15.623874 6484 consensus_peers.cc:487] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 -> Peer 04c7cfc64ea34fb388160a3e504c41cd (127.0.98.60:44061): Couldn't send request to peer 04c7cfc64ea34fb388160a3e504c41cd. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.60:44061: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 00:18:15.624429 7061 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2be6c69cb34f46079b9c5c5c45ad8ed1 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.625178 7247 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.625277 7247 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:15.625326 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 2be6c69cb34f46079b9c5c5c45ad8ed1 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.625191 7061 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.625938 7138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.626019 7138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:18:15.626019 7061 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:15.631599 7153 master.cc:552] Master@127.0.98.60:44061 shutdown complete.
I20260620 00:18:15.631673 7153 master_runner.cc:305] Clearing existing system tablet
I20260620 00:18:15.631820 7153 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:15.631883 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:15.632581 7153 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:18:15.633102 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.633260 7153 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:15.633323 7153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:18:15.633349 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal
uuid: "04c7cfc64ea34fb388160a3e504c41cd"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "9f2f9814cf00042df7a715e91eed1bcd"
server_key_iv: "acd9641253d943c910007b0bb7da6d4e"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.634150 7153 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:15.635632 7153 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260620 00:18:15.635713 7153 log.cc:1196] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260620 00:18:15.635910 7153 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: Deleting consensus metadata
I20260620 00:18:15.636214 7153 master_runner.cc:315] Copying system tablet from 127.0.98.62:34867
I20260620 00:18:15.636948 7153 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:34867
I20260620 00:18:15.643841 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.637379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40776 (local address 127.0.98.62:34867)
0620 00:18:15.637429 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.637431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.637499 (+ 68us) server_negotiation.cc:407] Connection header received
0620 00:18:15.637572 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.637572 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.637580 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.637632 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.638053 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.638557 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.639671 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.639895 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.639953 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.641138 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.641140 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.641141 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.641148 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.642640 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.643052 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.643054 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.643054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.643111 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.643312 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.643313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.643313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.643466 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.643521 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.643704 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.643774 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:18:15.643884 7257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.637285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:40776)
0620 00:18:15.637413 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.637425 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.637509 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.637645 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.637646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.637847 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.637904 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.637906 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.638650 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.638651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.639572 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.639574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.639975 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.639976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.640036 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.640602 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.640607 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.641044 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.642723 (+ 1679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.642724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.642725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.642952 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.643181 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.643182 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.643183 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.643250 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.643531 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.643532 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.643594 (+ 62us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.643677 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.643778 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":86,"threads_started":1}
I20260620 00:18:15.644376 6520 tablet_copy_service.cc:140] P b84405cafd7744b7b86c604bc3aa29a3: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 04c7cfc64ea34fb388160a3e504c41cd ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40776)
I20260620 00:18:15.644438 6520 tablet_copy_service.cc:161] P b84405cafd7744b7b86c604bc3aa29a3: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 04c7cfc64ea34fb388160a3e504c41cd at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40776: session id = 04c7cfc64ea34fb388160a3e504c41cd-00000000000000000000000000000000
I20260620 00:18:15.644917 6520 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: Tablet Copy: opened 0 blocks and 1 log segments
I20260620 00:18:15.645450 7153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260620 00:18:15.646456 7153 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: tablet copy: Starting download of 0 data blocks...
I20260620 00:18:15.646656 7153 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: tablet copy: Starting download of 1 WAL segments...
I20260620 00:18:15.647415 7153 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260620 00:18:15.648247 6520 tablet_copy_service.cc:342] P b84405cafd7744b7b86c604bc3aa29a3: Request end of tablet copy session 04c7cfc64ea34fb388160a3e504c41cd-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:40776
I20260620 00:18:15.648307 6520 tablet_copy_service.cc:433] P b84405cafd7744b7b86c604bc3aa29a3: ending tablet copy session 04c7cfc64ea34fb388160a3e504c41cd-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 04c7cfc64ea34fb388160a3e504c41cd
I20260620 00:18:15.650558 7153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:15.651858 7258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:15.651960 7259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:15.652190 7261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.652271 7153 server_base.cc:1034] running on GCE node
I20260620 00:18:15.652390 7153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:15.652572 7153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:15.653699 7153 hybrid_clock.cc:648] HybridClock initialized: now 1781914695653691 us; error 27 us; skew 500 ppm
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:15.655242 7153 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260620 00:18:15.655546 7153 webserver.cc:460] Webserver started at http://127.0.98.60:33487/ using document root <none> and password file <none>
I20260620 00:18:15.655666 7153 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:15.655717 7153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:15.656373 7153 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:15.656893 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.657052 7153 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:15.657111 7153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal
uuid: "04c7cfc64ea34fb388160a3e504c41cd"
format_stamp: "Formatted at 2026-06-20 00:18:15 on dist-test-slave-r81b"
server_key: "9f2f9814cf00042df7a715e91eed1bcd"
server_key_iv: "acd9641253d943c910007b0bb7da6d4e"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.657171 7153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:15.681797 7153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:15.683802 7153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:15.686939 7153 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:44061
I20260620 00:18:15.687194 7320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:44061 every 8 connection(s)
I20260620 00:18:15.687330 7153 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:18:15.688060 7321 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:18:15.688522 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: Bootstrap starting.
I20260620 00:18:15.690896 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:18:15.691139 7321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: Bootstrap complete.
I20260620 00:18:15.691376 7321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } }
I20260620 00:18:15.691470 7321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04c7cfc64ea34fb388160a3e504c41cd, State: Initialized, Role: LEARNER
I20260620 00:18:15.691514 7321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } }
I20260620 00:18:15.691730 7321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:15.691730 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:15.691855 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: This master's current role is: LEARNER
I20260620 00:18:15.693286 7335 catalog_manager.cc:1259] Loaded cluster ID: ffe755da79b54db7922d4b1a25f4001d
I20260620 00:18:15.693361 7335 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: loading cluster ID for follower catalog manager: success
I20260620 00:18:15.694420 7335 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: acquiring CA information for follower catalog manager: success
I20260620 00:18:15.694725 7335 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:18:15.719532 6691 heartbeater.cc:503] Master 127.0.98.62:34867 was elected leader, sending a full tablet report...
I20260620 00:18:15.728340 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.721098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40016 (local address 127.0.98.60:44061)
0620 00:18:15.721249 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.721251 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.721257 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:18:15.721277 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.721278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.721287 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.721339 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.722085 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.722780 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.723816 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.724016 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.724066 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.725663 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.725664 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.725664 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.725671 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.727113 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.727515 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.727516 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.727517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.727567 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.727778 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.727778 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.727778 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.727882 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.727983 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.728099 (+ 116us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.728238 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":83,"threads_started":1}
I20260620 00:18:15.728835 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 6562
I20260620 00:18:15.735589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45141
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:34867
--webserver_port=45915
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34867,127.0.98.61:36989
--tserver_master_addrs=127.0.98.62:34867,127.0.98.61:36989,127.0.98.60:44061 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:15.818100 7337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:15.818365 7337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:15.818406 7337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:15.819567 7337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:15.819641 7337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:15.819687 7337 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:15.821260 7337 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45141
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=45915
--tserver_master_addrs=127.0.98.62:34867,127.0.98.61:36989,127.0.98.60:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:15.821543 7337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:15.821784 7337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:15.826983 7342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:15.827042 7343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.827098 7337 server_base.cc:1034] running on GCE node
W20260620 00:18:15.827237 7345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.827457 7337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:15.827700 7337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:15.828881 7337 hybrid_clock.cc:648] HybridClock initialized: now 1781914695828872 us; error 36 us; skew 500 ppm
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:15.831656 7337 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:18:15.832221 7337 webserver.cc:460] Webserver started at http://127.0.98.1:45915/ using document root <none> and password file <none>
I20260620 00:18:15.832445 7337 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:15.832528 7337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:15.833937 7337 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:15.834769 7352 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.834986 7337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:15.835098 7337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "fe5e2f41487944fea605bbd3e80a0e43"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "0cec679ab530e3ff426e94c5060c638f"
server_key_iv: "d42f85b9abbe3a9ccd0c01d516b49c67"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.835512 7337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:15.852397 7337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:15.854882 7337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:15.855095 7337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:15.855357 7337 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:15.855681 7337 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:15.855738 7337 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.855777 7337 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:15.855808 7337 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.862085 7337 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45141
I20260620 00:18:15.862130 7465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45141 every 8 connection(s)
I20260620 00:18:15.862545 7337 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:18:15.870774 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 7337
I20260620 00:18:15.870935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 6696
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:15.875406 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.863295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50181 (local address 127.0.98.60:44061)
0620 00:18:15.863346 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.863348 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.863778 (+ 430us) server_negotiation.cc:407] Connection header received
0620 00:18:15.864911 (+ 1133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.864912 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.864926 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.864957 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.865700 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.866457 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.868321 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.868514 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.868593 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.872012 (+ 3419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.872014 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.872014 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.872022 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.873742 (+ 1720us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.874222 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.874224 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.874224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.874288 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.874543 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.874544 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.874545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.874705 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.874774 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.875134 (+ 360us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.875256 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:18:15.875766 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.863318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59467 (local address 127.0.98.62:34867)
0620 00:18:15.863365 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.863368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.863587 (+ 219us) server_negotiation.cc:407] Connection header received
0620 00:18:15.864895 (+ 1308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.864896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.864909 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.864964 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.865694 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.866406 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.867720 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.867999 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.868064 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.872894 (+ 4830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.872896 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.872897 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.872906 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.874606 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.874963 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.874964 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.874964 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.875026 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.875222 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.875223 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.875224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.875339 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.875410 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.875584 (+ 174us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.875668 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:18:15.875994 7472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.863290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.98.1:50181)
0620 00:18:15.863710 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.863719 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.864805 (+ 1086us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.865061 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.865062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.865370 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.865574 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.865582 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.866556 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.866556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.867799 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.867801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.868595 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.868596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.868710 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.869435 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.869446 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.871896 (+ 2450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.873852 (+ 1956us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.873859 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.873862 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.874114 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.874356 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.874357 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.874357 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.874473 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.874766 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.874773 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.874965 (+ 192us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.875503 (+ 538us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.875702 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":131,"thread_start_us":92,"threads_started":1}
I20260620 00:18:15.876178 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.863289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.1:59467)
0620 00:18:15.863454 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.863498 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.864784 (+ 1286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.865044 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.865047 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.865367 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.865582 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.865584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.866538 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.866538 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.867623 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.867625 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.868076 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.868077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.868712 (+ 635us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.869438 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.869447 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.872788 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.874691 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.874691 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.874692 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.874883 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.875089 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.875090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.875090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.875145 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.875412 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.875413 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.875457 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.875890 (+ 433us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.875997 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":109,"thread_start_us":96,"threads_started":1}
I20260620 00:18:15.876660 7466 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44061
I20260620 00:18:15.876765 7466 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:15.876820 7467 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34867
I20260620 00:18:15.876850 7467 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:15.876964 7466 heartbeater.cc:511] Master 127.0.98.60:44061 requested a full tablet report, sending...
I20260620 00:18:15.876963 7467 heartbeater.cc:511] Master 127.0.98.62:34867 requested a full tablet report, sending...
I20260620 00:18:15.877373 6499 ts_manager.cc:194] Re-registered known tserver with Master: fe5e2f41487944fea605bbd3e80a0e43 (127.0.98.1:45141)
I20260620 00:18:15.877454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40151
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:34867
--webserver_port=36323
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34867,127.0.98.61:36989
--tserver_master_addrs=127.0.98.62:34867,127.0.98.61:36989,127.0.98.60:44061 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:18:15.877733 7285 ts_manager.cc:194] Registered new tserver with Master: fe5e2f41487944fea605bbd3e80a0e43 (127.0.98.1:45141)
I20260620 00:18:15.877871 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59467
I20260620 00:18:15.879981 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.863775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40715 (local address 127.0.98.61:36989)
0620 00:18:15.863819 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:15.863820 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:15.863829 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:15.864910 (+ 1081us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:15.864911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:15.864924 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:15.864957 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:15.865708 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.866463 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.867880 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.868222 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.868274 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.877063 (+ 8789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:15.877065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:15.877066 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:15.877073 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:15.878769 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.879160 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.879161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.879161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.879219 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:15.879437 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:15.879438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:15.879439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:15.879563 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:15.879634 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:15.879810 (+ 176us) server_negotiation.cc:299] Negotiation successful
0620 00:18:15.879889 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:18:15.880002 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:15.863588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.98.1:40715)
0620 00:18:15.863776 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:15.863785 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:15.864805 (+ 1020us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:15.865061 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:15.865062 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:15.865417 (+ 355us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:15.865622 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.865624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.866556 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.866556 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.867799 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:15.867801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:15.868591 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:15.868592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:15.868726 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:15.869473 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:15.869478 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:15.876965 (+ 7487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:15.878911 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.878917 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.878918 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.879075 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.879295 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:15.879296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:15.879296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:15.879357 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:15.879636 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:15.879637 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:15.879692 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 00:18:15.879770 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 00:18:15.879891 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"mutex_wait_us":194,"thread_start_us":61,"threads_started":1}
I20260620 00:18:15.880347 7468 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36989
I20260620 00:18:15.880403 7468 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:15.880499 7468 heartbeater.cc:511] Master 127.0.98.61:36989 requested a full tablet report, sending...
I20260620 00:18:15.880864 7099 ts_manager.cc:194] Registered new tserver with Master: fe5e2f41487944fea605bbd3e80a0e43 (127.0.98.1:45141)
W20260620 00:18:15.962419 7473 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:15.962678 7473 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:15.962721 7473 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:15.963882 7473 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:15.963949 7473 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:15.963996 7473 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:15.965512 7473 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40151
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36323
--tserver_master_addrs=127.0.98.62:34867,127.0.98.61:36989,127.0.98.60:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7473
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:15.965790 7473 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:15.966024 7473 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:15.971285 7481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:15.971285 7478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:15.971352 7479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:15.971662 7473 server_base.cc:1034] running on GCE node
I20260620 00:18:15.971933 7473 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:15.972262 7473 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:15.973481 7473 hybrid_clock.cc:648] HybridClock initialized: now 1781914695973467 us; error 32 us; skew 500 ppm
Jun 20 00:18:15 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:15.976302 7473 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:18:15.976773 7473 webserver.cc:460] Webserver started at http://127.0.98.2:36323/ using document root <none> and password file <none>
I20260620 00:18:15.977026 7473 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:15.977129 7473 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:15.978500 7473 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:15.979097 7488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.979297 7473 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.979415 7473 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "c78bd803c43a4eee831fdba42b624364"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "162ed18ebf042a1199c14c369c82baad"
server_key_iv: "8edf0f70bf1ae3a9a59ef64b36c8d7b0"
server_key_version: "encryptionkey@0"
I20260620 00:18:15.979781 7473 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:15.995517 7473 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:15.998016 7473 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:15.998191 7473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:15.998432 7473 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:15.998739 7473 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:15.998795 7473 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:15.998837 7473 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:15.998865 7473 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:16.005239 7473 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40151
I20260620 00:18:16.005295 7601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40151 every 8 connection(s)
I20260620 00:18:16.005681 7473 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:16.012991 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 7473
I20260620 00:18:16.013150 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 6830
Jun 20 00:18:16 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Jun 20 00:18:16 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 00:18:16 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914695, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:18:16.018997 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.006513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55993 (local address 127.0.98.61:36989)
0620 00:18:16.006588 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.006590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.006863 (+ 273us) server_negotiation.cc:407] Connection header received
0620 00:18:16.008314 (+ 1451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.008316 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.008328 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.008362 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.009072 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.009922 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.011218 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.011434 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.011519 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.015264 (+ 3745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.015265 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.015266 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.015273 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.017197 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.017910 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.017911 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.017912 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.017970 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.018197 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.018198 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.018199 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.018343 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.018386 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.018765 (+ 379us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.018853 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:18:16.018997 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.006882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40697 (local address 127.0.98.60:44061)
0620 00:18:16.006932 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.006934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.007003 (+ 69us) server_negotiation.cc:407] Connection header received
0620 00:18:16.008213 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.008215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.008228 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.008287 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.009073 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.009936 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.011238 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.011550 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.011618 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.015028 (+ 3410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.015030 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.015030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.015039 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.017128 (+ 2089us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.017802 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.017804 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.017804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.017867 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.018217 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.018218 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.018218 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.018363 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.018414 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.018770 (+ 356us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.018862 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:18:16.019227 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.007008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35969 (local address 127.0.98.62:34867)
0620 00:18:16.007075 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.007078 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.007088 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:16.008171 (+ 1083us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.008172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.008204 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.008262 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.009062 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.009599 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.010613 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.010825 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.010913 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.015029 (+ 4116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.015030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.015030 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.015039 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.017866 (+ 2827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.018315 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.018316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.018316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.018372 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.018590 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.018590 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.018591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.018739 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.018785 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.019006 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.019084 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:18:16.019546 7607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.006497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.98.2:55993)
0620 00:18:16.006703 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.006773 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.008216 (+ 1443us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.008440 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.008441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.008781 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:16.008956 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.008963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.010033 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.010034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.011115 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.011117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.011526 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.011527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.011604 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.012064 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:16.012069 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:16.015153 (+ 3084us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:16.017286 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.017287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.017288 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.017705 (+ 417us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.018046 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.018047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.018047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.018126 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.018430 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:16.018435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:16.018604 (+ 169us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.019098 (+ 494us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.019291 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":125,"thread_start_us":124,"threads_started":1}
I20260620 00:18:16.019548 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38931
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:34867
--webserver_port=38199
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40797
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:34867,127.0.98.61:36989
--tserver_master_addrs=127.0.98.62:34867,127.0.98.61:36989,127.0.98.60:44061 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:18:16.019567 7606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.006496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.2:35969)
0620 00:18:16.006915 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.006926 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.008019 (+ 1093us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.008348 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.008351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.008745 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:16.008954 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.008959 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.009724 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.009725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.010520 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.010522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.010923 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.010924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.011049 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.011671 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:16.011682 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:16.014909 (+ 3227us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:16.017953 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.017954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.017954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.018228 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.018453 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.018454 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.018454 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.018512 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.018836 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:16.018836 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:16.018888 (+ 52us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.019352 (+ 464us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.019460 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"mutex_wait_us":3,"thread_start_us":114,"threads_started":1}
I20260620 00:18:16.020078 7603 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34867
I20260620 00:18:16.020221 7603 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:16.020243 7604 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36989
I20260620 00:18:16.020290 7604 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:16.020417 7603 heartbeater.cc:511] Master 127.0.98.62:34867 requested a full tablet report, sending...
I20260620 00:18:16.020419 7604 heartbeater.cc:511] Master 127.0.98.61:36989 requested a full tablet report, sending...
I20260620 00:18:16.020816 7608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.006813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.98.2:40697)
0620 00:18:16.006935 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.006946 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.008046 (+ 1100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.008350 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.008353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.008791 (+ 438us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:16.008956 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.008963 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.010033 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.010034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.011111 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.011113 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.011688 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.011689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.011784 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.012288 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:16.012293 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:16.014903 (+ 2610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:16.017229 (+ 2326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.017237 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.017240 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.017699 (+ 459us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.017953 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.017954 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.017954 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.018121 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.018465 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:16.018465 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:16.018604 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.020524 (+ 1920us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.020643 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"mutex_wait_us":51,"thread_start_us":67,"threads_started":1}
I20260620 00:18:16.020828 6499 ts_manager.cc:194] Re-registered known tserver with Master: c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151)
I20260620 00:18:16.021167 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35969
I20260620 00:18:16.021548 7099 ts_manager.cc:194] Registered new tserver with Master: c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151)
I20260620 00:18:16.021843 7602 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44061
I20260620 00:18:16.021947 7602 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:16.022079 7602 heartbeater.cc:511] Master 127.0.98.60:44061 requested a full tablet report, sending...
I20260620 00:18:16.022372 7285 ts_manager.cc:194] Registered new tserver with Master: c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151)
I20260620 00:18:16.089402 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.086378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.0.1:40028)
0620 00:18:16.086457 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.086472 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.086552 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.086733 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.086734 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.086735 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:16.086783 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.086785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.087689 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.087690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.088471 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.088473 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.089221 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.089222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.089265 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.089268 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.089309 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.089342 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260620 00:18:16.089426 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.086493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.0.98.60:44061)
0620 00:18:16.086567 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.086569 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.086596 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:18:16.086625 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.086626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.086648 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.086715 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:16.086853 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.087592 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.088553 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.089121 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.089201 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.089270 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.089315 (+ 45us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.089355 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260620 00:18:16.089829 7295 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260620 00:18:16.089996 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b84405cafd7744b7b86c604bc3aa29a3. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:16.090102 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: This master's current role is: LEARNER
W20260620 00:18:16.103631 7609 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:16.103847 7609 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:16.103871 7609 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:16.105093 7609 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:16.105140 7609 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:16.105170 7609 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:16.106734 7609 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40797
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38199
--tserver_master_addrs=127.0.98.62:34867,127.0.98.61:36989,127.0.98.60:44061
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7609
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:16.106927 7609 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:16.107115 7609 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:16.112332 7617 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:16.112342 7614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:16.112378 7615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:16.112511 7609 server_base.cc:1034] running on GCE node
I20260620 00:18:16.112785 7609 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:16.113022 7609 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:16.114174 7609 hybrid_clock.cc:648] HybridClock initialized: now 1781914696114150 us; error 39 us; skew 500 ppm
Jun 20 00:18:16 dist-test-slave-r81b krb5kdc[6455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:16.116740 7609 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:18:16.117233 7609 webserver.cc:460] Webserver started at http://127.0.98.3:38199/ using document root <none> and password file <none>
I20260620 00:18:16.117467 7609 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:16.117545 7609 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:16.118867 7609 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:16.119475 7624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:16.119710 7609 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:18:16.119829 7609 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "47f818f1a0f14277b5fb1a1a88ce472b"
format_stamp: "Formatted at 2026-06-20 00:18:14 on dist-test-slave-r81b"
server_key: "a88c629fb8a1258af7baeb53806e8687"
server_key_iv: "acda027f63978d7bb651a67e3b437529"
server_key_version: "encryptionkey@0"
I20260620 00:18:16.120290 7609 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:16.138357 7609 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:16.140578 7609 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:16.140744 7609 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:16.141011 7609 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:16.141323 7609 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:16.141358 7609 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:16.141381 7609 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:16.141394 7609 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:16.147114 7609 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38931
I20260620 00:18:16.147192 7737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38931 every 8 connection(s)
I20260620 00:18:16.147553 7609 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:18:16 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Jun 20 00:18:16 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260620 00:18:16.155218 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 7609
Jun 20 00:18:16 dist-test-slave-r81b krb5kdc[6455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914696, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:16.159008 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.148820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52267 (local address 127.0.98.61:36989)
0620 00:18:16.148867 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.148869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.148877 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:18:16.149439 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.149440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.149453 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.149534 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.150360 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.150859 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.151983 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.152256 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.152340 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.154945 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.154946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.154947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.154954 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.156899 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.157814 (+ 915us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.157815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.157816 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.157875 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.158170 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.158171 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.158171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.158313 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.158361 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.158807 (+ 446us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.158890 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:18:16.159415 6540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.148300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:60159 (local address 127.0.98.62:34867)
0620 00:18:16.148355 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.148357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.148601 (+ 244us) server_negotiation.cc:407] Connection header received
0620 00:18:16.149732 (+ 1131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.149734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.149750 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.149800 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.150375 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.151140 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.152707 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.152946 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.153037 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.156299 (+ 3262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.156301 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.156301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.156308 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.158024 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.158523 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.158524 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.158524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.158582 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.158820 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.158821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.158822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.158944 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.159000 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.159251 (+ 251us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.159327 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:18:16.159605 7743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.148325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.98.3:52267)
0620 00:18:16.148744 (+ 419us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.148753 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.149245 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.149619 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.149623 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.150051 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:16.150263 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.150272 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.150973 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.150974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.151892 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.151894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.152402 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.152403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.152559 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.153145 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:16.153153 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:16.154835 (+ 1682us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:16.157371 (+ 2536us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.157379 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.157381 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.157712 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.157950 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.157951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.157951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.158093 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.158412 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:16.158418 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:16.158611 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.159155 (+ 544us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.159339 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":388,"mutex_wait_us":4,"thread_start_us":89,"threads_started":1}
I20260620 00:18:16.159674 7744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.148602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.98.3:52001)
0620 00:18:16.148799 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.148810 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.149277 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.149637 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.149638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.150087 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:16.150265 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.150271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.151227 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.151227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.152235 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.152236 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.152661 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.152662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.152708 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.153362 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:16.153366 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:16.155803 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:16.158097 (+ 2294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.158098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.158099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.158490 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.158709 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.158710 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.158710 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.158766 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.159043 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:16.159043 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:16.159084 (+ 41us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.159156 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.159551 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":320,"thread_start_us":90,"threads_started":1}
I20260620 00:18:16.160622 7742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.148325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.3:60159)
0620 00:18:16.148485 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.148529 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.149651 (+ 1122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.149814 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.149815 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.150095 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:16.150265 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.150271 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.151227 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.151227 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.152229 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.152231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.153039 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.153040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.153111 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.153959 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:16.153963 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:16.156185 (+ 2222us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:16.158115 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.158116 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.158116 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.158452 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.158650 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:16.158650 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:16.158651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:16.158750 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:16.159027 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:16.159028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:16.159073 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.159155 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.160301 (+ 1146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":1032,"thread_start_us":92,"threads_started":1}
I20260620 00:18:16.160748 7740 heartbeater.cc:347] Connected to a master server at 127.0.98.61:36989
I20260620 00:18:16.160843 7740 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:16.160981 7739 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34867
I20260620 00:18:16.161026 7739 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:16.161044 7740 heartbeater.cc:511] Master 127.0.98.61:36989 requested a full tablet report, sending...
I20260620 00:18:16.161075 7739 heartbeater.cc:511] Master 127.0.98.62:34867 requested a full tablet report, sending...
I20260620 00:18:16.161468 6499 ts_manager.cc:194] Re-registered known tserver with Master: 47f818f1a0f14277b5fb1a1a88ce472b (127.0.98.3:38931)
I20260620 00:18:16.161489 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.148679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52001 (local address 127.0.98.60:44061)
0620 00:18:16.148727 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.148729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.148887 (+ 158us) server_negotiation.cc:407] Connection header received
0620 00:18:16.149499 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.149501 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.149515 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.149573 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.150374 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.151127 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.152320 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.152581 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.152650 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.156131 (+ 3481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.156156 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.156157 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.156165 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.157990 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.158568 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.158569 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.158569 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.158640 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.158828 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.158829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.158829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.158946 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.159017 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.161295 (+ 2278us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.161410 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:18:16.161600 7099 ts_manager.cc:194] Registered new tserver with Master: 47f818f1a0f14277b5fb1a1a88ce472b (127.0.98.3:38931)
I20260620 00:18:16.161947 6499 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:60159
I20260620 00:18:16.162338 7738 heartbeater.cc:347] Connected to a master server at 127.0.98.60:44061
I20260620 00:18:16.162393 7738 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:16.162454 7738 heartbeater.cc:511] Master 127.0.98.60:44061 requested a full tablet report, sending...
I20260620 00:18:16.162726 7285 ts_manager.cc:194] Registered new tserver with Master: 47f818f1a0f14277b5fb1a1a88ce472b (127.0.98.3:38931)
I20260620 00:18:16.166884 7749 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.157163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40780 (local address 127.0.98.62:34867)
0620 00:18:16.157310 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.157312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.157504 (+ 192us) server_negotiation.cc:407] Connection header received
0620 00:18:16.157530 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.157531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.157543 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.157602 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.158569 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.159371 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.160509 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.160793 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.160856 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.163575 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.163576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.163577 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.163590 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.165401 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.165773 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.165774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.165775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.165833 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.166061 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.166062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.166063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.166207 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.166645 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.166686 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.166771 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260620 00:18:16.169596 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.161827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43948 (local address 127.0.98.61:36989)
0620 00:18:16.161864 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.161865 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.162068 (+ 203us) server_negotiation.cc:407] Connection header received
0620 00:18:16.162146 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.162146 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.162156 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.162200 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.162880 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.163342 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.164356 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.164609 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.164677 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.166023 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.166024 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.166025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.166032 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.167872 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.168697 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.168698 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.168699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.168754 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.168961 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.168962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.168962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.169128 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.169221 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.169394 (+ 173us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.169480 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:18:16.170907 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.162099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40038 (local address 127.0.98.60:44061)
0620 00:18:16.162248 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.162249 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.163667 (+ 1418us) server_negotiation.cc:407] Connection header received
0620 00:18:16.163724 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.163725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.163735 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.163789 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.164486 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.165204 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.166233 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.166494 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.166585 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.168386 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.168387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.168387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.168394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.169880 (+ 1486us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.170250 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.170251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.170252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.170306 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.170504 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.170504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.170504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.170593 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.170659 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.170792 (+ 133us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.170851 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260620 00:18:16.169016 6499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:18:16.171398 6499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:16.172189 7754 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: true } } }
I20260620 00:18:16.172283 7754 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:16.173492 7755 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: attempting to promote NON_VOTER 04c7cfc64ea34fb388160a3e504c41cd to VOTER
I20260620 00:18:16.173622 7755 consensus_queue.cc:237] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } }
I20260620 00:18:16.174206 7295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 1 LEARNER]: Refusing update from remote peer b84405cafd7744b7b86c604bc3aa29a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 00:18:16.174214 7108 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Refusing update from remote peer b84405cafd7744b7b86c604bc3aa29a3: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260620 00:18:16.174492 7757 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 00:18:16.174571 7754 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260620 00:18:16.175115 7757 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 04c7cfc64ea34fb388160a3e504c41cd (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:16.175297 7108 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 04c7cfc64ea34fb388160a3e504c41cd (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:16.175297 7295 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 04c7cfc64ea34fb388160a3e504c41cd (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:16.175990 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:16.176052 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:16.176112 7755 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:16.176229 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:16.176302 7755 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:16.176314 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:18:16.175956 7757 sys_catalog.cc:455] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:16.176410 7757 sys_catalog.cc:458] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:16.176687 7764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b84405cafd7744b7b86c604bc3aa29a3" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:16.176777 7764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:18:16.181246 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.177315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36824 (local address 127.0.98.2:40151)
0620 00:18:16.177480 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.177484 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.177494 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:16.177540 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.177542 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.177556 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.177647 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:16.177760 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.179279 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.180224 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.180868 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.180954 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.181040 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.181112 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.181146 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260620 00:18:16.181282 7059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.177009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38931 (local address 127.0.0.1:53156)
0620 00:18:16.177054 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.177065 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.177104 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.177843 (+ 739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.177844 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.177845 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:16.177875 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.177876 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.178865 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.178866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.179959 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.179961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.181023 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.181024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.181091 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.181096 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.181127 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.181167 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260620 00:18:16.181305 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.177087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40151 (local address 127.0.0.1:36824)
0620 00:18:16.177203 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.177212 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.177252 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.177644 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.177645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.177647 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:16.177698 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.177700 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.179365 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.179366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.180080 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.180081 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.180969 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.180970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.181033 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.181039 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.181096 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.181145 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":92,"threads_started":1}
I20260620 00:18:16.181701 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.177155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53156 (local address 127.0.98.3:38931)
0620 00:18:16.177688 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.177692 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.177700 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:18:16.177735 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.177736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.177750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.177944 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:16.177998 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.178775 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.180055 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.180920 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.181467 (+ 547us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.181539 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.181563 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.181622 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":71,"threads_started":1}
I20260620 00:18:16.182093 7536 tablet_service.cc:1467] Processing CreateTablet for tablet 33da3541d510489c8a4214ac7c0c8dac (DEFAULT_TABLE table=test-table [id=c21b568c81cc463382171453dabafc81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:16.182241 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.177105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45141 (local address 127.0.0.1:33228)
0620 00:18:16.177498 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.177507 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.177565 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.177803 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.177804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.177806 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:16.177852 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.177854 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.179156 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.179157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.180255 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.180257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.181233 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.181234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.181293 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.182046 (+ 753us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.182127 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.182175 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":81,"threads_started":1}
I20260620 00:18:16.182266 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.177307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.0.98.1:45141)
0620 00:18:16.177475 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.177480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.177551 (+ 71us) server_negotiation.cc:407] Connection header received
0620 00:18:16.177642 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.177645 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.177661 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.177736 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:16.177938 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.179072 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.180348 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.181043 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.181124 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.181199 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.182144 (+ 945us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.182188 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260620 00:18:16.182480 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33da3541d510489c8a4214ac7c0c8dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:16.182456 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 33da3541d510489c8a4214ac7c0c8dac (DEFAULT_TABLE table=test-table [id=c21b568c81cc463382171453dabafc81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:16.182790 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33da3541d510489c8a4214ac7c0c8dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:16.182860 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 33da3541d510489c8a4214ac7c0c8dac (DEFAULT_TABLE table=test-table [id=c21b568c81cc463382171453dabafc81]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:16.183162 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33da3541d510489c8a4214ac7c0c8dac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:16.186352 7770 tablet_bootstrap.cc:492] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: Bootstrap starting.
I20260620 00:18:16.186929 7770 tablet_bootstrap.cc:654] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:16.187186 7770 log.cc:824] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:16.187446 7771 tablet_bootstrap.cc:492] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: Bootstrap starting.
I20260620 00:18:16.187894 7770 tablet_bootstrap.cc:492] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: No bootstrap required, opened a new log
I20260620 00:18:16.187973 7770 ts_tablet_manager.cc:1397] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:18:16.188046 7771 tablet_bootstrap.cc:654] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:16.188289 7772 tablet_bootstrap.cc:492] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: Bootstrap starting.
I20260620 00:18:16.188406 7771 log.cc:824] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:16.188846 7772 tablet_bootstrap.cc:654] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:16.189159 7772 log.cc:824] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:16.189196 7771 tablet_bootstrap.cc:492] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: No bootstrap required, opened a new log
I20260620 00:18:16.189247 7771 ts_tablet_manager.cc:1397] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:18:16.189950 7772 tablet_bootstrap.cc:492] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: No bootstrap required, opened a new log
I20260620 00:18:16.190012 7772 ts_tablet_manager.cc:1397] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:18:16.190421 7770 raft_consensus.cc:348] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } }
I20260620 00:18:16.190574 7770 raft_consensus.cc:374] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:16.190615 7770 raft_consensus.cc:729] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f818f1a0f14277b5fb1a1a88ce472b, State: Initialized, Role: FOLLOWER
I20260620 00:18:16.190747 7770 consensus_queue.cc:260] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } }
I20260620 00:18:16.191009 7770 ts_tablet_manager.cc:1428] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:18:16.191231 7739 heartbeater.cc:503] Master 127.0.98.62:34867 was elected leader, sending a full tablet report...
I20260620 00:18:16.191248 7771 raft_consensus.cc:348] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } }
I20260620 00:18:16.191483 7771 raft_consensus.cc:374] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:16.191525 7771 raft_consensus.cc:729] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78bd803c43a4eee831fdba42b624364, State: Initialized, Role: FOLLOWER
I20260620 00:18:16.191733 7771 consensus_queue.cc:260] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } }
I20260620 00:18:16.191977 7771 ts_tablet_manager.cc:1428] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:18:16.192296 7603 heartbeater.cc:503] Master 127.0.98.62:34867 was elected leader, sending a full tablet report...
I20260620 00:18:16.193466 7772 raft_consensus.cc:348] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } }
I20260620 00:18:16.193631 7772 raft_consensus.cc:374] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:16.193668 7772 raft_consensus.cc:729] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5e2f41487944fea605bbd3e80a0e43, State: Initialized, Role: FOLLOWER
I20260620 00:18:16.193786 7772 consensus_queue.cc:260] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } }
I20260620 00:18:16.194049 7772 ts_tablet_manager.cc:1428] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 00:18:16.194114 7467 heartbeater.cc:503] Master 127.0.98.62:34867 was elected leader, sending a full tablet report...
W20260620 00:18:16.256536 7605 tablet.cc:2369] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:18:16.324383 7778 raft_consensus.cc:482] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:16.324542 7778 raft_consensus.cc:504] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } }
I20260620 00:18:16.324950 7778 leader_election.cc:288] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f818f1a0f14277b5fb1a1a88ce472b (127.0.98.3:38931), c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151)
I20260620 00:18:16.327826 7470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.325107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38931 (local address 127.0.98.1:48493)
0620 00:18:16.325163 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.325175 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.325220 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.325427 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.325428 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.325430 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:16.325508 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.325510 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.326163 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.326165 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.326974 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.326976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.327555 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.327555 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.327638 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.327641 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.327710 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.327752 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 00:18:16.327826 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.325167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48493 (local address 127.0.98.3:38931)
0620 00:18:16.325224 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.325226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.325235 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:16.325310 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.325311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.325329 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.325421 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:16.325592 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.326071 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.327094 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.327451 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.327550 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.327560 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.327716 (+ 156us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.327752 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:18:16.327955 7471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.325233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40151 (local address 127.0.98.1:59619)
0620 00:18:16.325277 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:16.325287 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:16.325322 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:16.325476 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:16.325477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:16.325478 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:16.325549 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.325550 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.326439 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.326441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.327133 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:16.327135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.327682 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.327683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:16.327751 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.327753 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:18:16.327823 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 00:18:16.327865 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 00:18:16.327977 7769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.325284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59619 (local address 127.0.98.2:40151)
0620 00:18:16.325329 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.325331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.325342 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:16.325372 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.325373 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.325384 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.325470 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:16.325619 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.326352 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.327221 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.327592 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.327682 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.327692 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.327842 (+ 150us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.327912 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:18:16.328347 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33da3541d510489c8a4214ac7c0c8dac" candidate_uuid: "fe5e2f41487944fea605bbd3e80a0e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" is_pre_election: true
I20260620 00:18:16.328408 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33da3541d510489c8a4214ac7c0c8dac" candidate_uuid: "fe5e2f41487944fea605bbd3e80a0e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c78bd803c43a4eee831fdba42b624364" is_pre_election: true
I20260620 00:18:16.328498 7692 raft_consensus.cc:2454] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe5e2f41487944fea605bbd3e80a0e43 in term 0.
I20260620 00:18:16.328610 7556 raft_consensus.cc:2454] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe5e2f41487944fea605bbd3e80a0e43 in term 0.
I20260620 00:18:16.328784 7356 leader_election.cc:302] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f818f1a0f14277b5fb1a1a88ce472b, fe5e2f41487944fea605bbd3e80a0e43; no voters:
I20260620 00:18:16.328905 7778 raft_consensus.cc:2781] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:16.328979 7778 raft_consensus.cc:482] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:16.329020 7778 raft_consensus.cc:3037] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:16.329881 7778 raft_consensus.cc:504] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } }
I20260620 00:18:16.330070 7778 leader_election.cc:288] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [CANDIDATE]: Term 1 election: Requested vote from peers 47f818f1a0f14277b5fb1a1a88ce472b (127.0.98.3:38931), c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151)
I20260620 00:18:16.330194 7692 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33da3541d510489c8a4214ac7c0c8dac" candidate_uuid: "fe5e2f41487944fea605bbd3e80a0e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f818f1a0f14277b5fb1a1a88ce472b"
I20260620 00:18:16.330246 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33da3541d510489c8a4214ac7c0c8dac" candidate_uuid: "fe5e2f41487944fea605bbd3e80a0e43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c78bd803c43a4eee831fdba42b624364"
I20260620 00:18:16.330312 7556 raft_consensus.cc:3037] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:16.330312 7692 raft_consensus.cc:3037] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:16.331156 7692 raft_consensus.cc:2454] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe5e2f41487944fea605bbd3e80a0e43 in term 1.
I20260620 00:18:16.331195 7556 raft_consensus.cc:2454] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe5e2f41487944fea605bbd3e80a0e43 in term 1.
I20260620 00:18:16.331331 7356 leader_election.cc:302] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f818f1a0f14277b5fb1a1a88ce472b, fe5e2f41487944fea605bbd3e80a0e43; no voters:
I20260620 00:18:16.331451 7778 raft_consensus.cc:2781] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:16.331630 7778 raft_consensus.cc:686] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 1 LEADER]: Becoming Leader. State: Replica: fe5e2f41487944fea605bbd3e80a0e43, State: Running, Role: LEADER
I20260620 00:18:16.333606 7778 consensus_queue.cc:237] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } }
I20260620 00:18:16.334869 6499 catalog_manager.cc:5495] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe5e2f41487944fea605bbd3e80a0e43 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fe5e2f41487944fea605bbd3e80a0e43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } health_report { overall_health: UNKNOWN } } }
I20260620 00:18:16.361428 7768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.359084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33244 (local address 127.0.98.1:45141)
0620 00:18:16.359142 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.359145 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.359154 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:16.359199 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.359200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.359224 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.359288 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:16.359416 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.359892 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.360736 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.360916 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.360969 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.361154 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:16.361254 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:16.361314 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.361350 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.361367 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:18:16.363408 7556 raft_consensus.cc:1261] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [term 1 FOLLOWER]: Refusing update from remote peer fe5e2f41487944fea605bbd3e80a0e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:16.363476 7692 raft_consensus.cc:1261] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [term 1 FOLLOWER]: Refusing update from remote peer fe5e2f41487944fea605bbd3e80a0e43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:16.363770 7778 consensus_queue.cc:1035] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:18:16.363950 7469 tablet.cc:2369] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:18:16.363958 7779 consensus_queue.cc:1035] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:16.367370 7784 mvcc.cc:205] Tried to move back new op lower bound from 7298722596301508608 to 7298722596183445504. Current Snapshot: MvccSnapshot[applied={T|T < 7298722596301508608}]
I20260620 00:18:16.368062 7783 mvcc.cc:205] Tried to move back new op lower bound from 7298722596301508608 to 7298722596183445504. Current Snapshot: MvccSnapshot[applied={T|T < 7298722596301508608}]
I20260620 00:18:16.371043 6499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:16.371150 6499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40780:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:16.372916 6499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b84405cafd7744b7b86c604bc3aa29a3: Sending DeleteTablet for 3 replicas of tablet 33da3541d510489c8a4214ac7c0c8dac
I20260620 00:18:16.374426 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 6468
I20260620 00:18:16.376096 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet 33da3541d510489c8a4214ac7c0c8dac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:16 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36824
I20260620 00:18:16.376482 7792 tablet_replica.cc:330] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: stopping tablet replica
I20260620 00:18:16.376624 7792 raft_consensus.cc:2229] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:16.376736 7792 raft_consensus.cc:2258] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:16.381258 7792 ts_tablet_manager.cc:1905] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 00:18:16.383208 7603 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34867 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.98.62:34867 (error 108)
I20260620 00:18:16.383409 7792 ts_tablet_manager.cc:1918] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:16.383544 7792 log.cc:1196] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/33da3541d510489c8a4214ac7c0c8dac
I20260620 00:18:16.383826 7792 ts_tablet_manager.cc:1939] T 33da3541d510489c8a4214ac7c0c8dac P c78bd803c43a4eee831fdba42b624364: Deleting consensus metadata
W20260620 00:18:16.384191 7492 connection.cc:441] server connection from 127.0.0.1:36824 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:36824 (request call id 1) could send its response
I20260620 00:18:16.392688 7336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.385112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40046 (local address 127.0.98.60:44061)
0620 00:18:16.385154 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.385156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.385255 (+ 99us) server_negotiation.cc:407] Connection header received
0620 00:18:16.385332 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.385333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.385343 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.385393 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.385930 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.386507 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.387494 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.387788 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.387825 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.389500 (+ 1675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.389501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.389502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.389509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.391259 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.391702 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.391703 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.391704 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.391776 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.391991 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.391992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.391992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.392170 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.392260 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.392499 (+ 239us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.392580 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:18:16.393527 7150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:16.384847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43956 (local address 127.0.98.61:36989)
0620 00:18:16.384905 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:16.384907 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:16.385103 (+ 196us) server_negotiation.cc:407] Connection header received
0620 00:18:16.385207 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:16.385208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:16.385219 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:16.385250 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:16.386357 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.387069 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.388024 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:16.388696 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:16.388745 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:16.390487 (+ 1742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:16.390489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:16.390489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:16.390496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:16.392228 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.392611 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.392612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.392613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.392663 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:16.392932 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:16.392933 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:16.392934 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:16.393069 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:16.393139 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:16.393369 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 00:18:16.393448 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
W20260620 00:18:16.778573 7354 consensus_peers.cc:487] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 -> Peer c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151): Couldn't send request to peer c78bd803c43a4eee831fdba42b624364. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 33da3541d510489c8a4214ac7c0c8dac. This is attempt 1: this message will repeat every 5th retry.
I20260620 00:18:17.368834 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.368535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.1:34281)
0620 00:18:17.368702 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.368774 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
W20260620 00:18:17.369081 7467 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34867 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
I20260620 00:18:17.369499 7800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.369316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.3:54811)
0620 00:18:17.369411 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.369456 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
W20260620 00:18:17.369737 7739 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:34867 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
I20260620 00:18:17.383848 7801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.383553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.2:44831)
0620 00:18:17.383687 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.383794 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":57,"threads_started":1}
I20260620 00:18:17.397993 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.397843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.1:38913)
0620 00:18:17.397901 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.397947 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260620 00:18:17.819271 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.819111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.1:39779)
0620 00:18:17.819180 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.819220 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 00:18:17.872862 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b84405cafd7744b7b86c604bc3aa29a3)
I20260620 00:18:17.873018 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } }
I20260620 00:18:17.873268 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b84405cafd7744b7b86c604bc3aa29a3 (127.0.98.62:34867), 04c7cfc64ea34fb388160a3e504c41cd (127.0.98.60:44061)
I20260620 00:18:17.873649 7808 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b84405cafd7744b7b86c604bc3aa29a3)
I20260620 00:18:17.873680 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.873403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:41258)
0620 00:18:17.873568 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.873630 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
W20260620 00:18:17.873804 7084 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
I20260620 00:18:17.873915 7808 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } }
I20260620 00:18:17.874190 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.874107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:41262)
0620 00:18:17.874142 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.874165 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260620 00:18:17.874255 7084 leader_election.cc:334] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b84405cafd7744b7b86c604bc3aa29a3 (127.0.98.62:34867): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
I20260620 00:18:17.874383 7808 leader_election.cc:288] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b84405cafd7744b7b86c604bc3aa29a3 (127.0.98.62:34867), 2be6c69cb34f46079b9c5c5c45ad8ed1 (127.0.98.61:36989)
I20260620 00:18:17.874569 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.874296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:41274)
0620 00:18:17.874474 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.874528 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":46,"threads_started":1}
W20260620 00:18:17.874681 7270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
I20260620 00:18:17.875124 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.875028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:41276)
0620 00:18:17.875071 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.875094 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:18:17.875223 7270 leader_election.cc:334] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b84405cafd7744b7b86c604bc3aa29a3 (127.0.98.62:34867): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
I20260620 00:18:17.877270 7810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.873730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40048 (local address 127.0.98.60:44061)
0620 00:18:17.873834 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:17.873836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:17.873846 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:17.873875 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:17.873875 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:17.873887 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:17.873962 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:17.874179 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.874997 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.876290 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.876945 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.877027 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.877039 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:17.877198 (+ 159us) server_negotiation.cc:299] Negotiation successful
0620 00:18:17.877229 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
I20260620 00:18:17.877350 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.873595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:44061 (local address 127.0.0.1:40048)
0620 00:18:17.873708 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.873719 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:17.873781 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:17.874020 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:17.874021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:17.874023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:17.874097 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.874099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.875083 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.875084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.876208 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.876210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.877033 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.877034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.877084 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.877088 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:17.877182 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 00:18:17.877265 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":88,"mutex_wait_us":36,"thread_start_us":49,"threads_started":1}
I20260620 00:18:17.877540 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04c7cfc64ea34fb388160a3e504c41cd" is_pre_election: true
I20260620 00:18:17.877655 7295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2be6c69cb34f46079b9c5c5c45ad8ed1 in term 1.
I20260620 00:18:17.877828 7083 leader_election.cc:302] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c7cfc64ea34fb388160a3e504c41cd, 2be6c69cb34f46079b9c5c5c45ad8ed1; no voters: b84405cafd7744b7b86c604bc3aa29a3
I20260620 00:18:17.877918 7806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260620 00:18:17.877959 7806 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b84405cafd7744b7b86c604bc3aa29a3)
I20260620 00:18:17.877990 7806 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:18:17.878048 7812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.874312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36989 (local address 127.0.0.1:43960)
0620 00:18:17.874511 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.874521 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:17.874584 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:17.874930 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:17.874931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:17.874933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:17.874976 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.874978 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.875910 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.875911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.877062 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.877063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.877789 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.877790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.877858 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.877861 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:17.877934 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:18:17.877976 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":48,"threads_started":1}
I20260620 00:18:17.878057 7814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.874368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43960 (local address 127.0.98.61:36989)
0620 00:18:17.874489 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:17.874491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:17.874590 (+ 99us) server_negotiation.cc:407] Connection header received
0620 00:18:17.874780 (+ 190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:17.874781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:17.874791 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:17.874860 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:17.875064 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.875819 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.877145 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.877713 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.877788 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.877800 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:17.877941 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 00:18:17.877981 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":82,"threads_started":1}
I20260620 00:18:17.878253 7108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "04c7cfc64ea34fb388160a3e504c41cd" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" is_pre_election: true
I20260620 00:18:17.878793 7806 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } }
I20260620 00:18:17.878947 7806 leader_election.cc:288] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [CANDIDATE]: Term 2 election: Requested vote from peers b84405cafd7744b7b86c604bc3aa29a3 (127.0.98.62:34867), 04c7cfc64ea34fb388160a3e504c41cd (127.0.98.60:44061)
I20260620 00:18:17.878973 7108 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 04c7cfc64ea34fb388160a3e504c41cd in current term 2: Already voted for candidate 2be6c69cb34f46079b9c5c5c45ad8ed1 in this term.
I20260620 00:18:17.879086 7295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "04c7cfc64ea34fb388160a3e504c41cd"
I20260620 00:18:17.879150 7295 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:18:17.879240 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.879045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:41294)
0620 00:18:17.879187 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.879217 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260620 00:18:17.879241 7269 leader_election.cc:302] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04c7cfc64ea34fb388160a3e504c41cd; no voters: 2be6c69cb34f46079b9c5c5c45ad8ed1, b84405cafd7744b7b86c604bc3aa29a3
I20260620 00:18:17.879565 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.879464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:41298)
0620 00:18:17.879519 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.879546 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 00:18:17.879640 7084 leader_election.cc:334] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b84405cafd7744b7b86c604bc3aa29a3 (127.0.98.62:34867): Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
I20260620 00:18:17.879953 7295 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2be6c69cb34f46079b9c5c5c45ad8ed1 in term 2.
I20260620 00:18:17.880033 7808 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:18:17.880131 7083 leader_election.cc:302] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04c7cfc64ea34fb388160a3e504c41cd, 2be6c69cb34f46079b9c5c5c45ad8ed1; no voters: b84405cafd7744b7b86c604bc3aa29a3
I20260620 00:18:17.880239 7806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:18:17.880393 7806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [term 2 LEADER]: Becoming Leader. State: Replica: 2be6c69cb34f46079b9c5c5c45ad8ed1, State: Running, Role: LEADER
I20260620 00:18:17.880496 7806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } }
I20260620 00:18:17.880841 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2be6c69cb34f46079b9c5c5c45ad8ed1. Latest consensus state: current_term: 2 leader_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:17.880942 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:17.881135 7818 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:17.881487 7818 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:17.881538 7818 catalog_manager.cc:1259] Loaded cluster ID: ffe755da79b54db7922d4b1a25f4001d
I20260620 00:18:17.881556 7818 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:17.881680 7818 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:17.881731 7818 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Loaded TSK: 0
I20260620 00:18:17.881829 7818 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:18:17.936085 7099 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:18:17.937981 7099 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:17.939209 7295 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [term 2 FOLLOWER]: Refusing update from remote peer 2be6c69cb34f46079b9c5c5c45ad8ed1: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260620 00:18:17.939246 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.939071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:41312)
0620 00:18:17.939166 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.939206 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260620 00:18:17.939658 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.939522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.0.1:41318)
0620 00:18:17.939559 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.939587 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 00:18:17.939606 7816 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260620 00:18:17.939750 7084 consensus_peers.cc:487] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 -> Peer b84405cafd7744b7b86c604bc3aa29a3 (127.0.98.62:34867): Couldn't send request to peer b84405cafd7744b7b86c604bc3aa29a3. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260620 00:18:17.940361 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2be6c69cb34f46079b9c5c5c45ad8ed1. Latest consensus state: current_term: 2 leader_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:17.940481 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:18:17.940968 7816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:17.941080 7816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:17.941205 7808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b84405cafd7744b7b86c604bc3aa29a3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34867 } } peers { permanent_uuid: "2be6c69cb34f46079b9c5c5c45ad8ed1" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36989 } attrs { promote: false } } peers { permanent_uuid: "04c7cfc64ea34fb388160a3e504c41cd" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 44061 } attrs { promote: false } } }
I20260620 00:18:17.941300 7808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04c7cfc64ea34fb388160a3e504c41cd [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:18:17.941545 7820 mvcc.cc:205] Tried to move back new op lower bound from 7298722602756657152 to 7298722602519166976. Current Snapshot: MvccSnapshot[applied={T|T < 7298722602756657152 or (T in {7298722602756657152})}]
I20260620 00:18:17.946094 7807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.942789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38931 (local address 127.0.0.1:53160)
0620 00:18:17.942823 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.942835 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:17.942884 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:17.943438 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:17.943439 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:17.943442 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:17.943483 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.943485 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.944412 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.944413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.945204 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.945206 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.945881 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.945882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.945947 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.945951 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:17.945979 (+ 28us) client_negotiation.cc:240] Negotiation successful
0620 00:18:17.946020 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":54}
I20260620 00:18:17.946496 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.942800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36840 (local address 127.0.98.2:40151)
0620 00:18:17.942902 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:17.942904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:17.942912 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:18:17.942940 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:17.942941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:17.942950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:17.943075 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:17.943232 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.944030 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.945573 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.946140 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.946217 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.946234 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:17.946374 (+ 140us) server_negotiation.cc:299] Negotiation successful
0620 00:18:17.946414 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":51,"threads_started":1}
I20260620 00:18:17.946506 7809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.942740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40151 (local address 127.0.0.1:36840)
0620 00:18:17.942775 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.942787 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:17.942830 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:17.943094 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:17.943095 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:17.943097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:17.943153 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.943155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.944119 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.944120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.945486 (+ 1366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.945489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.946227 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.946228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.946293 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.946298 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:18:17.946362 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 00:18:17.946401 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":28}
I20260620 00:18:17.946715 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.942905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45141 (local address 127.0.0.1:33256)
0620 00:18:17.943087 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.943097 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:17.943137 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:17.943594 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:17.943595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:17.943597 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:17.943642 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.943644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.944586 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.944587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.945782 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:17.945784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.946493 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.946494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:17.946559 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.946565 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:18:17.946600 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 00:18:17.946646 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":125,"threads_started":1}
I20260620 00:18:17.946784 7536 tablet_service.cc:1467] Processing CreateTablet for tablet 024af0fc67b34beab63eb27d7a338385 (DEFAULT_TABLE table=test-table [id=618cd8e70930455baac85011491b63a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:17.946941 7536 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024af0fc67b34beab63eb27d7a338385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:17.947059 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.943174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.0.98.3:38931)
0620 00:18:17.943307 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:17.943309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:17.943316 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:18:17.943340 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:17.943340 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:17.943349 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:17.943554 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:17.943592 (+ 38us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.944333 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.945304 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.945790 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.946488 (+ 698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.946514 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:17.946534 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 00:18:17.946961 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260620 00:18:17.947067 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.943281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33256 (local address 127.0.98.1:45141)
0620 00:18:17.943389 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:17.943391 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:17.943399 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:18:17.943467 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:17.943469 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:17.943480 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:17.943718 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:17.943764 (+ 46us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.944478 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.945881 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:17.946399 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:17.946883 (+ 484us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:17.946899 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:17.946922 (+ 23us) server_negotiation.cc:299] Negotiation successful
0620 00:18:17.946983 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":48,"threads_started":1}
I20260620 00:18:17.947285 7672 tablet_service.cc:1467] Processing CreateTablet for tablet 024af0fc67b34beab63eb27d7a338385 (DEFAULT_TABLE table=test-table [id=618cd8e70930455baac85011491b63a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:17.947413 7672 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024af0fc67b34beab63eb27d7a338385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:17.948369 7400 tablet_service.cc:1467] Processing CreateTablet for tablet 024af0fc67b34beab63eb27d7a338385 (DEFAULT_TABLE table=test-table [id=618cd8e70930455baac85011491b63a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:17.948518 7400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 024af0fc67b34beab63eb27d7a338385. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:17.948673 7828 tablet_bootstrap.cc:492] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: Bootstrap starting.
I20260620 00:18:17.949010 7828 tablet_bootstrap.cc:654] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:17.949057 7829 tablet_bootstrap.cc:492] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: Bootstrap starting.
I20260620 00:18:17.949388 7829 tablet_bootstrap.cc:654] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:17.949605 7828 tablet_bootstrap.cc:492] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: No bootstrap required, opened a new log
I20260620 00:18:17.949647 7828 ts_tablet_manager.cc:1397] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:17.949757 7828 raft_consensus.cc:348] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } }
I20260620 00:18:17.949818 7828 raft_consensus.cc:374] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:17.949841 7828 raft_consensus.cc:729] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c78bd803c43a4eee831fdba42b624364, State: Initialized, Role: FOLLOWER
I20260620 00:18:17.949898 7828 consensus_queue.cc:260] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } }
I20260620 00:18:17.949998 7829 tablet_bootstrap.cc:492] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: No bootstrap required, opened a new log
I20260620 00:18:17.950026 7832 tablet_bootstrap.cc:492] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43: Bootstrap starting.
I20260620 00:18:17.950045 7829 ts_tablet_manager.cc:1397] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:17.950043 7828 ts_tablet_manager.cc:1428] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:17.950215 7829 raft_consensus.cc:348] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } }
I20260620 00:18:17.950294 7829 raft_consensus.cc:374] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:17.950322 7829 raft_consensus.cc:729] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f818f1a0f14277b5fb1a1a88ce472b, State: Initialized, Role: FOLLOWER
I20260620 00:18:17.950374 7832 tablet_bootstrap.cc:654] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:17.950407 7829 consensus_queue.cc:260] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } }
I20260620 00:18:17.950603 7829 ts_tablet_manager.cc:1428] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:17.950944 7832 tablet_bootstrap.cc:492] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43: No bootstrap required, opened a new log
I20260620 00:18:17.950999 7832 ts_tablet_manager.cc:1397] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:17.951143 7832 raft_consensus.cc:348] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } }
I20260620 00:18:17.951231 7832 raft_consensus.cc:374] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:17.951261 7832 raft_consensus.cc:729] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe5e2f41487944fea605bbd3e80a0e43, State: Initialized, Role: FOLLOWER
I20260620 00:18:17.951265 7604 heartbeater.cc:503] Master 127.0.98.61:36989 was elected leader, sending a full tablet report...
I20260620 00:18:17.951315 7832 consensus_queue.cc:260] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } }
I20260620 00:18:17.951439 7832 ts_tablet_manager.cc:1428] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:17.951653 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.951263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.3:55745)
0620 00:18:17.951543 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.951581 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":94,"threads_started":1}
I20260620 00:18:17.951957 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.951843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.1:50629)
0620 00:18:17.951886 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.951922 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 00:18:17.951983 7740 heartbeater.cc:503] Master 127.0.98.61:36989 was elected leader, sending a full tablet report...
I20260620 00:18:17.952076 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.951986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.3:40637)
0620 00:18:17.952023 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.952047 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 00:18:17.952446 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 33da3541d510489c8a4214ac7c0c8dac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:16 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53160
I20260620 00:18:17.952699 7838 tablet_replica.cc:330] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: stopping tablet replica
I20260620 00:18:17.952821 7838 raft_consensus.cc:2229] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:17.952890 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.950698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.2:35761)
0620 00:18:17.951124 (+ 426us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.951342 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":75,"threads_started":1}
I20260620 00:18:17.952952 7838 raft_consensus.cc:2258] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:17.953189 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:17.953113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.2:38481)
0620 00:18:17.953144 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:17.953168 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260620 00:18:17.953254 7838 ts_tablet_manager.cc:1905] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:17.954674 7838 ts_tablet_manager.cc:1918] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:17.954738 7838 log.cc:1196] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/33da3541d510489c8a4214ac7c0c8dac
I20260620 00:18:17.955025 7838 ts_tablet_manager.cc:1939] T 33da3541d510489c8a4214ac7c0c8dac P 47f818f1a0f14277b5fb1a1a88ce472b: Deleting consensus metadata
I20260620 00:18:17.955406 7086 catalog_manager.cc:4841] TS 47f818f1a0f14277b5fb1a1a88ce472b (127.0.98.3:38931): tablet 33da3541d510489c8a4214ac7c0c8dac (table test-table [id=c21b568c81cc463382171453dabafc81]) successfully deleted
W20260620 00:18:18.149701 7741 tablet.cc:2369] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:18:18.181437 7834 raft_consensus.cc:482] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:18.181568 7834 raft_consensus.cc:504] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } }
I20260620 00:18:18.181977 7834 leader_election.cc:288] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151), fe5e2f41487944fea605bbd3e80a0e43 (127.0.98.1:45141)
I20260620 00:18:18.186126 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.182209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40151 (local address 127.0.98.3:49023)
0620 00:18:18.182266 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.182278 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:18.182340 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:18.182556 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:18.182557 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:18.182561 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:18.182661 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:18.182663 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.183663 (+ 1000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.183664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:18.185054 (+ 1390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:18.185057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.185850 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.185851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:18.185918 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:18.185921 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:18.185973 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:18:18.186021 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34,"mutex_wait_us":13}
I20260620 00:18:18.186195 7827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.182278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39541 (local address 127.0.98.1:45141)
0620 00:18:18.182332 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:18.182334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:18.182619 (+ 285us) server_negotiation.cc:407] Connection header received
0620 00:18:18.182645 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:18.182646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:18.182657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:18.182699 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:18.182838 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.183589 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.185160 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.185731 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.185815 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:18.185827 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:18.186063 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 00:18:18.186117 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:18:18.186196 7824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.182328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49023 (local address 127.0.98.2:40151)
0620 00:18:18.182377 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:18.182380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:18.182391 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:18.182437 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:18.182438 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:18.182452 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:18.182528 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:18.182752 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.183562 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.185160 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.185731 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.185815 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:18.185827 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:18.186059 (+ 232us) server_negotiation.cc:299] Negotiation successful
0620 00:18:18.186123 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:18:18.186223 7839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.182214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45141 (local address 127.0.98.3:39541)
0620 00:18:18.182499 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.182527 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:18.182583 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:18.182718 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:18.182719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:18.182721 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:18.182768 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:18.182770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.183672 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.183673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:18.185054 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:18.185057 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.185825 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.185826 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:18.185901 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:18.185906 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:18:18.185970 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 00:18:18.186021 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":162,"thread_start_us":144,"threads_started":1}
I20260620 00:18:18.186441 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024af0fc67b34beab63eb27d7a338385" candidate_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c78bd803c43a4eee831fdba42b624364" is_pre_election: true
I20260620 00:18:18.186554 7556 raft_consensus.cc:2454] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f818f1a0f14277b5fb1a1a88ce472b in term 0.
I20260620 00:18:18.186517 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024af0fc67b34beab63eb27d7a338385" candidate_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe5e2f41487944fea605bbd3e80a0e43" is_pre_election: true
I20260620 00:18:18.186666 7420 raft_consensus.cc:2454] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f818f1a0f14277b5fb1a1a88ce472b in term 0.
I20260620 00:18:18.186880 7628 leader_election.cc:302] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f818f1a0f14277b5fb1a1a88ce472b, fe5e2f41487944fea605bbd3e80a0e43; no voters:
I20260620 00:18:18.187053 7834 raft_consensus.cc:2781] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:18.187105 7834 raft_consensus.cc:482] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:18.187137 7834 raft_consensus.cc:3037] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:18.187911 7834 raft_consensus.cc:504] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } }
I20260620 00:18:18.188072 7834 leader_election.cc:288] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [CANDIDATE]: Term 1 election: Requested vote from peers c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151), fe5e2f41487944fea605bbd3e80a0e43 (127.0.98.1:45141)
I20260620 00:18:18.188257 7420 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024af0fc67b34beab63eb27d7a338385" candidate_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fe5e2f41487944fea605bbd3e80a0e43"
I20260620 00:18:18.188274 7556 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "024af0fc67b34beab63eb27d7a338385" candidate_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c78bd803c43a4eee831fdba42b624364"
I20260620 00:18:18.188344 7420 raft_consensus.cc:3037] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:18.188361 7556 raft_consensus.cc:3037] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:18.188968 7556 raft_consensus.cc:2454] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f818f1a0f14277b5fb1a1a88ce472b in term 1.
I20260620 00:18:18.189030 7420 raft_consensus.cc:2454] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f818f1a0f14277b5fb1a1a88ce472b in term 1.
I20260620 00:18:18.189162 7626 leader_election.cc:302] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f818f1a0f14277b5fb1a1a88ce472b, c78bd803c43a4eee831fdba42b624364; no voters:
I20260620 00:18:18.189314 7834 raft_consensus.cc:2781] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:18.189513 7834 raft_consensus.cc:686] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 1 LEADER]: Becoming Leader. State: Replica: 47f818f1a0f14277b5fb1a1a88ce472b, State: Running, Role: LEADER
I20260620 00:18:18.189602 7834 consensus_queue.cc:237] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } }
I20260620 00:18:18.189802 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.189704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.3:57455)
0620 00:18:18.189738 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.189771 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 00:18:18.190389 7099 catalog_manager.cc:5495] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f818f1a0f14277b5fb1a1a88ce472b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f818f1a0f14277b5fb1a1a88ce472b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38931 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 } health_report { overall_health: UNKNOWN } } }
I20260620 00:18:18.231994 7826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.229512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.0.98.3:38931)
0620 00:18:18.229565 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:18.229567 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:18.229576 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:18.229618 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:18.229620 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:18.229645 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:18.229723 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:18.229909 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.230387 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.231167 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:18.231366 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:18.231446 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:18.231634 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:18.231743 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:18.231848 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:18.231874 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 00:18:18.231898 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:18:18.233505 7420 raft_consensus.cc:1261] T 024af0fc67b34beab63eb27d7a338385 P fe5e2f41487944fea605bbd3e80a0e43 [term 1 FOLLOWER]: Refusing update from remote peer 47f818f1a0f14277b5fb1a1a88ce472b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:18.233523 7556 raft_consensus.cc:1261] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [term 1 FOLLOWER]: Refusing update from remote peer 47f818f1a0f14277b5fb1a1a88ce472b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:18.233840 7834 consensus_queue.cc:1035] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c78bd803c43a4eee831fdba42b624364" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40151 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:18.234015 7840 consensus_queue.cc:1035] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fe5e2f41487944fea605bbd3e80a0e43" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45141 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:18.234797 7468 heartbeater.cc:503] Master 127.0.98.61:36989 was elected leader, sending a full tablet report...
I20260620 00:18:18.235141 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.234979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.1:49275)
0620 00:18:18.235036 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.235075 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:18:18.235782 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 33da3541d510489c8a4214ac7c0c8dac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:16 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33256
I20260620 00:18:18.235284 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.235082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.2:53667)
0620 00:18:18.235147 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.235181 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 00:18:18.236081 7848 tablet_replica.cc:330] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: stopping tablet replica
I20260620 00:18:18.236228 7848 raft_consensus.cc:2229] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:18:18.236362 7848 raft_consensus.cc:2258] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:18.236755 7848 ts_tablet_manager.cc:1905] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:18.237000 7845 mvcc.cc:205] Tried to move back new op lower bound from 7298722603961106432 to 7298722603785162752. Current Snapshot: MvccSnapshot[applied={T|T < 7298722603961106432}]
I20260620 00:18:18.237461 7799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.237352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.1:42869)
0620 00:18:18.237395 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.237427 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 00:18:18.238241 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.238125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.3:42823)
0620 00:18:18.238170 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.238200 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 00:18:18.238523 7836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.237178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.2:54353)
0620 00:18:18.238410 (+ 1232us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.238490 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1213}
I20260620 00:18:18.238559 7837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.238467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34867 (local address 127.0.98.3:47497)
0620 00:18:18.238506 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.238530 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:34867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260620 00:18:18.239521 7848 ts_tablet_manager.cc:1918] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:18.239601 7848 log.cc:1196] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/33da3541d510489c8a4214ac7c0c8dac
I20260620 00:18:18.239954 7848 ts_tablet_manager.cc:1939] T 33da3541d510489c8a4214ac7c0c8dac P fe5e2f41487944fea605bbd3e80a0e43: Deleting consensus metadata
I20260620 00:18:18.240469 7086 catalog_manager.cc:4841] TS fe5e2f41487944fea605bbd3e80a0e43 (127.0.98.1:45141): tablet 33da3541d510489c8a4214ac7c0c8dac (table test-table [id=c21b568c81cc463382171453dabafc81]) successfully deleted
I20260620 00:18:18.241073 7099 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:18.241189 7099 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:18.242676 7099 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2be6c69cb34f46079b9c5c5c45ad8ed1: Sending DeleteTablet for 3 replicas of tablet 024af0fc67b34beab63eb27d7a338385
I20260620 00:18:18.243024 7536 tablet_service.cc:1514] Processing DeleteTablet for tablet 024af0fc67b34beab63eb27d7a338385 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:18 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:36840
I20260620 00:18:18.243321 7672 tablet_service.cc:1514] Processing DeleteTablet for tablet 024af0fc67b34beab63eb27d7a338385 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:18 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:53160
I20260620 00:18:18.243379 7400 tablet_service.cc:1514] Processing DeleteTablet for tablet 024af0fc67b34beab63eb27d7a338385 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:18 UTC) from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:33256
I20260620 00:18:18.243475 7855 tablet_replica.cc:330] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: stopping tablet replica
I20260620 00:18:18.243513 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 7337
I20260620 00:18:18.243537 7855 raft_consensus.cc:2229] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:18.243583 7855 raft_consensus.cc:2258] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:18.243582 7838 tablet_replica.cc:330] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: stopping tablet replica
I20260620 00:18:18.243640 7838 raft_consensus.cc:2229] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:18:18.243743 7838 raft_consensus.cc:2258] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:18.244733 7855 ts_tablet_manager.cc:1905] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:18.246308 7855 ts_tablet_manager.cc:1918] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:18.246359 7855 log.cc:1196] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/024af0fc67b34beab63eb27d7a338385
I20260620 00:18:18.246579 7855 ts_tablet_manager.cc:1939] T 024af0fc67b34beab63eb27d7a338385 P c78bd803c43a4eee831fdba42b624364: Deleting consensus metadata
I20260620 00:18:18.246973 7084 catalog_manager.cc:4841] TS c78bd803c43a4eee831fdba42b624364 (127.0.98.2:40151): tablet 024af0fc67b34beab63eb27d7a338385 (table test-table [id=618cd8e70930455baac85011491b63a0]) successfully deleted
I20260620 00:18:18.248502 7838 ts_tablet_manager.cc:1905] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:18.249972 7838 ts_tablet_manager.cc:1918] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:18.250038 7838 log.cc:1196] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/024af0fc67b34beab63eb27d7a338385
I20260620 00:18:18.250253 7838 ts_tablet_manager.cc:1939] T 024af0fc67b34beab63eb27d7a338385 P 47f818f1a0f14277b5fb1a1a88ce472b: Deleting consensus metadata
I20260620 00:18:18.250706 7086 catalog_manager.cc:4841] TS 47f818f1a0f14277b5fb1a1a88ce472b (127.0.98.3:38931): tablet 024af0fc67b34beab63eb27d7a338385 (table test-table [id=618cd8e70930455baac85011491b63a0]) successfully deleted
I20260620 00:18:18.251271 7825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:18.251159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45141 (local address 127.0.0.1:33270)
0620 00:18:18.251200 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:18.251233 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
I20260620 00:18:18.251454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 7473
W20260620 00:18:18.251518 7086 catalog_manager.cc:4584] TS fe5e2f41487944fea605bbd3e80a0e43 (127.0.98.1:45141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 024af0fc67b34beab63eb27d7a338385: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45141: connect: Connection refused (error 111)
I20260620 00:18:18.257704 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 7609
I20260620 00:18:18.262780 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 6964
I20260620 00:18:18.267555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 7153
2026-06-20T00:18:18Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (5047 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:18 dist-test-slave-r81b krb5kdc[7858](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:18 dist-test-slave-r81b krb5kdc[7858](info): set up 2 sockets
Jun 20 00:18:18 dist-test-slave-r81b krb5kdc[7858](info): commencing operation
krb5kdc: starting...
W20260620 00:18:19.308697 392 mini_kdc.cc:121] Time spent starting KDC: real 1.018s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:19 dist-test-slave-r81b krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:19Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260620 00:18:19.434832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46313
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:19.517402 7871 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:19.517673 7871 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:19.517711 7871 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:19.518947 7871 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:19.519018 7871 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260620 00:18:19.519042 7871 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:19.519064 7871 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:19.519109 7871 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:19.519136 7871 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:19.520725 7871 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46313
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7871
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:19.521016 7871 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:19.521250 7871 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:19.526955 7876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:19.526938 7879 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:19.526963 7877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:19.527207 7871 server_base.cc:1034] running on GCE node
I20260620 00:18:19.527514 7871 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:19.527840 7871 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:19.528987 7871 hybrid_clock.cc:648] HybridClock initialized: now 1781914699528971 us; error 56 us; skew 500 ppm
Jun 20 00:18:19 dist-test-slave-r81b krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914699, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:19.536751 7871 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260620 00:18:19.537310 7871 webserver.cc:460] Webserver started at http://127.0.98.62:38317/ using document root <none> and password file <none>
I20260620 00:18:19.537591 7871 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:19.537678 7871 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:19.537838 7871 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:19.538976 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "2daad448310d4c128e14a0753b386267"
format_stamp: "Formatted at 2026-06-20 00:18:19 on dist-test-slave-r81b"
server_key: "7d7952748f425f278770eadf5de50686"
server_key_iv: "59765cdeb8b7deafaddf368c2a55d9d7"
server_key_version: "encryptionkey@0"
I20260620 00:18:19.539347 7871 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "2daad448310d4c128e14a0753b386267"
format_stamp: "Formatted at 2026-06-20 00:18:19 on dist-test-slave-r81b"
server_key: "7d7952748f425f278770eadf5de50686"
server_key_iv: "59765cdeb8b7deafaddf368c2a55d9d7"
server_key_version: "encryptionkey@0"
I20260620 00:18:19.540988 7871 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:19.542042 7886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:19.542260 7871 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:19.542382 7871 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "2daad448310d4c128e14a0753b386267"
format_stamp: "Formatted at 2026-06-20 00:18:19 on dist-test-slave-r81b"
server_key: "7d7952748f425f278770eadf5de50686"
server_key_iv: "59765cdeb8b7deafaddf368c2a55d9d7"
server_key_version: "encryptionkey@0"
I20260620 00:18:19.542466 7871 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:19.565644 7871 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:19.567799 7871 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:19.567967 7871 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:19.572594 7871 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46313
I20260620 00:18:19.572638 7948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46313 every 8 connection(s)
I20260620 00:18:19.572968 7871 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:19.573611 7949 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:19.575951 7949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267: Bootstrap starting.
I20260620 00:18:19.576623 7949 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:19.576927 7949 log.cc:824] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:19.577692 7949 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267: No bootstrap required, opened a new log
I20260620 00:18:19.579806 7949 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daad448310d4c128e14a0753b386267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46313 } }
I20260620 00:18:19.579993 7949 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:19.580049 7949 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2daad448310d4c128e14a0753b386267, State: Initialized, Role: FOLLOWER
I20260620 00:18:19.580297 7949 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daad448310d4c128e14a0753b386267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46313 } }
I20260620 00:18:19.580374 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 7871
I20260620 00:18:19.580410 7949 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:19.580459 7949 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:19.580466 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:19.580509 7949 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:19.580668 392 external_mini_cluster.cc:1428] Setting key 5753785ea568750dad5ac0f577cf2cac
I20260620 00:18:19.581169 7949 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daad448310d4c128e14a0753b386267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46313 } }
I20260620 00:18:19.581281 7949 leader_election.cc:302] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2daad448310d4c128e14a0753b386267; no voters:
I20260620 00:18:19.581427 7949 leader_election.cc:288] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:19.581468 7954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:19.581621 7954 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [term 1 LEADER]: Becoming Leader. State: Replica: 2daad448310d4c128e14a0753b386267, State: Running, Role: LEADER
I20260620 00:18:19.581686 7949 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:19.581712 7954 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daad448310d4c128e14a0753b386267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46313 } }
I20260620 00:18:19.582056 7954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2daad448310d4c128e14a0753b386267. Latest consensus state: current_term: 1 leader_uuid: "2daad448310d4c128e14a0753b386267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daad448310d4c128e14a0753b386267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46313 } } }
I20260620 00:18:19.582142 7954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:19.582068 7955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2daad448310d4c128e14a0753b386267" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2daad448310d4c128e14a0753b386267" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46313 } } }
I20260620 00:18:19.582484 7955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:19.582494 7961 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:19.583003 7961 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:19.585168 7961 catalog_manager.cc:1347] Generated new cluster ID: da13a50afd2a406eb146fa1f2d888c3e
I20260620 00:18:19.585220 7961 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:18:19 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914699, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:19.592860 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:19.581199 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46556 (local address 127.0.98.62:46313)
0620 00:18:19.581438 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:19.581449 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:19.581486 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:18:19.581622 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:19.581630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:19.581694 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:19.582759 (+ 1065us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:19.582871 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.584310 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.585394 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.585986 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.586095 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:19.588970 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:19.588974 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:19.588974 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:19.588988 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:19.591104 (+ 2116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.591513 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.591526 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.591527 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.591644 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.591876 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.591877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.591877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.592209 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:19.592364 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:19.592501 (+ 137us) server_negotiation.cc:299] Negotiation successful
0620 00:18:19.592639 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":92,"threads_started":1}
I20260620 00:18:19.608172 7961 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:19.609009 7961 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:19.613420 7961 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267: Generated new TSK 0
I20260620 00:18:19.613628 7961 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260620 00:18:19.640089 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46313
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:19.730901 7975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:19.731124 7975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:19.731153 7975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:19.732570 7975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:19.732618 7975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:19.732641 7975 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:18:19.732661 7975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:19.734339 7975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:19.734632 7975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:19.734903 7975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:19.740437 7980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:19.740425 7983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:19.740425 7981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:19.740703 7975 server_base.cc:1034] running on GCE node
I20260620 00:18:19.740933 7975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:19.741187 7975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:19.742376 7975 hybrid_clock.cc:648] HybridClock initialized: now 1781914699742370 us; error 37 us; skew 500 ppm
Jun 20 00:18:19 dist-test-slave-r81b krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914699, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:19.745095 7975 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:18:19.745769 7975 webserver.cc:460] Webserver started at http://127.0.98.1:36725/ using document root <none> and password file <none>
I20260620 00:18:19.746070 7975 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:19.746137 7975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:19.746284 7975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:19.747466 7975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "8d87d9c06e364abda3c8cdc8fa885787"
format_stamp: "Formatted at 2026-06-20 00:18:19 on dist-test-slave-r81b"
server_key: "e4350956b29e4f748854d5dfc4060d0c"
server_key_iv: "b3adf41a403d2e2aa134a076fd9de1b1"
server_key_version: "encryptionkey@0"
I20260620 00:18:19.747836 7975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8d87d9c06e364abda3c8cdc8fa885787"
format_stamp: "Formatted at 2026-06-20 00:18:19 on dist-test-slave-r81b"
server_key: "e4350956b29e4f748854d5dfc4060d0c"
server_key_iv: "b3adf41a403d2e2aa134a076fd9de1b1"
server_key_version: "encryptionkey@0"
I20260620 00:18:19.749261 7975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:19.750119 7990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:19.750298 7975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:19.750399 7975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "8d87d9c06e364abda3c8cdc8fa885787"
format_stamp: "Formatted at 2026-06-20 00:18:19 on dist-test-slave-r81b"
server_key: "e4350956b29e4f748854d5dfc4060d0c"
server_key_iv: "b3adf41a403d2e2aa134a076fd9de1b1"
server_key_version: "encryptionkey@0"
I20260620 00:18:19.750483 7975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:19.765367 7975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:19.767659 7975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:19.767843 7975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:19.768574 7975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:19.768642 7975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:19.768680 7975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:19.768704 7975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:18:19 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914699, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:19.776126 7975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41723
I20260620 00:18:19.776300 8105 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41723 every 8 connection(s)
I20260620 00:18:19.776611 7975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:18:19.779100 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:19.769100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33515 (local address 127.0.98.62:46313)
0620 00:18:19.769181 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:19.769184 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:19.769348 (+ 164us) server_negotiation.cc:407] Connection header received
0620 00:18:19.769869 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:19.769870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:19.769884 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:19.769968 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:19.770690 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.771566 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.772882 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.773116 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.773223 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:19.775987 (+ 2764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:19.775989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:19.775989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:19.776004 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:19.777781 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.778191 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.778192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.778192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.778246 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.778519 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.778520 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.778520 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.778649 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:19.778710 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:19.778956 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 00:18:19.779036 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 00:18:19.779291 7999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:19.769060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46313 (local address 127.0.98.1:33515)
0620 00:18:19.769207 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:19.769249 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:19.769764 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:19.770040 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:19.770044 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:19.770333 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:19.770575 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:19.770580 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.771667 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.771668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:19.772768 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:19.772771 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.773245 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.773247 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:19.773442 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:19.774062 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:19.774074 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:19.775848 (+ 1774us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:19.777872 (+ 2024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:19.777884 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:19.777886 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:19.778117 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:19.778308 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:19.778308 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:19.778309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:19.778418 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:19.778713 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:19.778717 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:19.778841 (+ 124us) client_negotiation.cc:769] Sending connection context
0620 00:18:19.778927 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 00:18:19.779055 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260620 00:18:19.779817 8106 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46313
I20260620 00:18:19.779927 8106 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:19.780176 8106 heartbeater.cc:511] Master 127.0.98.62:46313 requested a full tablet report, sending...
I20260620 00:18:19.780733 7902 ts_manager.cc:194] Registered new tserver with Master: 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723)
I20260620 00:18:19.781765 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33515
I20260620 00:18:19.785686 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 7975
I20260620 00:18:19.785801 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:19.786029 392 external_mini_cluster.cc:1428] Setting key ce1f237c98b4655ea27efff5ee2c2726
I20260620 00:18:19.787770 8112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:19.780856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46313 (local address 127.0.98.1:43905)
0620 00:18:19.781027 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:19.781040 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:19.781111 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:19.781220 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:19.781221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:19.781415 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:19.781483 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:19.781484 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.782330 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.782332 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:19.783061 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:19.783064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.783449 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.783450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:19.783512 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:19.784076 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:19.784081 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:19.784492 (+ 411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:19.786445 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:19.786447 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:19.786447 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:19.786722 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:19.787014 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:19.787015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:19.787015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:19.787106 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:19.787487 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:19.787488 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:19.787533 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:18:19.787602 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:18:19.787692 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":35,"thread_start_us":93,"threads_started":1}
I20260620 00:18:19.787904 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:19.780901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43905 (local address 127.0.98.62:46313)
0620 00:18:19.780941 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:19.780943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:19.781127 (+ 184us) server_negotiation.cc:407] Connection header received
0620 00:18:19.781165 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:19.781166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:19.781175 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:19.781216 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:19.781555 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.782234 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.783133 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.783316 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.783438 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:19.784602 (+ 1164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:19.784603 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:19.784604 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:19.784611 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:19.786308 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.786823 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.786825 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.786825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.786929 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.787190 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.787191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.787192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.787377 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:19.787469 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:19.787653 (+ 184us) server_negotiation.cc:299] Negotiation successful
0620 00:18:19.787752 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260620 00:18:19.815922 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46313
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:19.905092 8115 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:19.905318 8115 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:19.905349 8115 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:19.906584 8115 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:19.906632 8115 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:19.906654 8115 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:18:19.906672 8115 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:19.908210 8115 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8115
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:19.908476 8115 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:19.908712 8115 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:19.913928 8121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:19.913903 8123 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:19.913923 8120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:19.913955 8115 server_base.cc:1034] running on GCE node
I20260620 00:18:19.914335 8115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:19.914590 8115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:19.915731 8115 hybrid_clock.cc:648] HybridClock initialized: now 1781914699915713 us; error 35 us; skew 500 ppm
Jun 20 00:18:19 dist-test-slave-r81b krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914699, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:19.918368 8115 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:18:19.918912 8115 webserver.cc:460] Webserver started at http://127.0.98.2:35605/ using document root <none> and password file <none>
I20260620 00:18:19.919174 8115 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:19.919245 8115 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:19.919395 8115 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:19.920575 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e"
format_stamp: "Formatted at 2026-06-20 00:18:19 on dist-test-slave-r81b"
server_key: "73a38a02daa630cd58a22c595d5224d0"
server_key_iv: "0994395af5a365874885db0326fbfb6e"
server_key_version: "encryptionkey@0"
I20260620 00:18:19.920959 8115 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e"
format_stamp: "Formatted at 2026-06-20 00:18:19 on dist-test-slave-r81b"
server_key: "73a38a02daa630cd58a22c595d5224d0"
server_key_iv: "0994395af5a365874885db0326fbfb6e"
server_key_version: "encryptionkey@0"
I20260620 00:18:19.922297 8115 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:19.923072 8130 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:19.923233 8115 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:19.923339 8115 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e"
format_stamp: "Formatted at 2026-06-20 00:18:19 on dist-test-slave-r81b"
server_key: "73a38a02daa630cd58a22c595d5224d0"
server_key_iv: "0994395af5a365874885db0326fbfb6e"
server_key_version: "encryptionkey@0"
I20260620 00:18:19.923417 8115 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:19.949489 8115 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:19.951556 8115 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:19.951730 8115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:19.952400 8115 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:19.952450 8115 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:19.952482 8115 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:19.952502 8115 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:18:19 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914699, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:19.960018 8115 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42547
I20260620 00:18:19.960162 8245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42547 every 8 connection(s)
I20260620 00:18:19.960453 8115 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:19.961689 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 8115
I20260620 00:18:19.961808 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:19.962075 392 external_mini_cluster.cc:1428] Setting key 5989a028f08c1ae77288067377780efa
I20260620 00:18:19.963550 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:19.952813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41923 (local address 127.0.98.62:46313)
0620 00:18:19.952881 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:19.952884 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:19.953172 (+ 288us) server_negotiation.cc:407] Connection header received
0620 00:18:19.953925 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:19.953927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:19.953938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:19.953981 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:19.954935 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.955432 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.956624 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.956821 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.956900 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:19.960274 (+ 3374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:19.960276 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:19.960276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:19.960285 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:19.961762 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.962318 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.962319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.962319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.962388 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.962656 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.962657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.962658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.962816 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:19.962956 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:19.963309 (+ 353us) server_negotiation.cc:299] Negotiation successful
0620 00:18:19.963408 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:18:19.963559 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:19.952814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46313 (local address 127.0.98.2:41923)
0620 00:18:19.953000 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:19.953056 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:19.953803 (+ 747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:19.954074 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:19.954080 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:19.954500 (+ 420us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:19.954816 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:19.954823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.955565 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.955566 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:19.956530 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:19.956532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.956906 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.956908 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:19.957063 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:19.957645 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:19.957657 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:19.960171 (+ 2514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:19.961895 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:19.961902 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:19.961904 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:19.962235 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:19.962461 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:19.962462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:19.962463 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:19.962582 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:19.962944 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:19.962948 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:19.963144 (+ 196us) client_negotiation.cc:769] Sending connection context
0620 00:18:19.963256 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:18:19.963391 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":85,"threads_started":1}
I20260620 00:18:19.964079 8246 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46313
I20260620 00:18:19.964207 8246 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:19.964425 8246 heartbeater.cc:511] Master 127.0.98.62:46313 requested a full tablet report, sending...
I20260620 00:18:19.964799 7902 ts_manager.cc:194] Registered new tserver with Master: a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547)
I20260620 00:18:19.965274 7902 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41923
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:18:19.971187 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:19.965267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44715 (local address 127.0.98.62:46313)
0620 00:18:19.965307 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:19.965309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:19.965462 (+ 153us) server_negotiation.cc:407] Connection header received
0620 00:18:19.965542 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:19.965543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:19.965554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:19.965612 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:19.965962 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.966437 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.967241 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.967447 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.967520 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:19.968542 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:19.968543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:19.968543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:19.968550 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:19.969925 (+ 1375us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.970289 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.970290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.970291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.970354 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:19.970554 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:19.970555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:19.970555 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:19.970765 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:19.970838 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:19.971016 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 00:18:19.971092 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:18:19.971202 8253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:19.965252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46313 (local address 127.0.98.2:44715)
0620 00:18:19.965400 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:19.965411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:19.965489 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:19.965615 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:19.965616 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:19.965816 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:19.965873 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:19.965875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.966530 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.966531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:19.967146 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:19.967148 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:19.967518 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:19.967519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:19.967582 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:19.968101 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:19.968106 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:19.968443 (+ 337us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:19.970025 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:19.970026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:19.970026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:19.970202 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:19.970422 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:19.970423 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:19.970423 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:19.970482 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:19.970839 (+ 357us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:19.970841 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:19.970886 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:18:19.970989 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 00:18:19.971093 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 00:18:19.988905 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46313
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33249
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:20.070708 8255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:20.070978 8255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:20.071025 8255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:20.072284 8255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:20.072362 8255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:20.072396 8255 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260620 00:18:20.072435 8255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:20.074025 8255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33249
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46313
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:20.074328 8255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:20.074581 8255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:20.079927 8261 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:20.079931 8263 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:20.079934 8260 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:20.079947 8255 server_base.cc:1034] running on GCE node
I20260620 00:18:20.080487 8255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:20.080776 8255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:20.081970 8255 hybrid_clock.cc:648] HybridClock initialized: now 1781914700081964 us; error 38 us; skew 500 ppm
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914700, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:20.085091 8255 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:18:20.085590 8255 webserver.cc:460] Webserver started at http://127.0.98.3:44851/ using document root <none> and password file <none>
I20260620 00:18:20.085848 8255 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:20.085987 8255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:20.086144 8255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:20.087251 8255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "1b7f02ab3881440abe79ebb295000a38"
format_stamp: "Formatted at 2026-06-20 00:18:20 on dist-test-slave-r81b"
server_key: "38051f71683f4a1fcd202683dc90ec20"
server_key_iv: "34e6dd34b56c6d848e634e4edb537b74"
server_key_version: "encryptionkey@0"
I20260620 00:18:20.087638 8255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1b7f02ab3881440abe79ebb295000a38"
format_stamp: "Formatted at 2026-06-20 00:18:20 on dist-test-slave-r81b"
server_key: "38051f71683f4a1fcd202683dc90ec20"
server_key_iv: "34e6dd34b56c6d848e634e4edb537b74"
server_key_version: "encryptionkey@0"
I20260620 00:18:20.089126 8255 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:20.089857 8270 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:20.090190 8255 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:20.090301 8255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "1b7f02ab3881440abe79ebb295000a38"
format_stamp: "Formatted at 2026-06-20 00:18:20 on dist-test-slave-r81b"
server_key: "38051f71683f4a1fcd202683dc90ec20"
server_key_iv: "34e6dd34b56c6d848e634e4edb537b74"
server_key_version: "encryptionkey@0"
I20260620 00:18:20.090399 8255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:20.106285 8255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:20.108776 8255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:20.108963 8255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:20.109508 8255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:20.109577 8255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:20.109627 8255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:20.109652 8255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914700, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:20.117040 8255 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35389
I20260620 00:18:20.117146 8385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35389 every 8 connection(s)
I20260620 00:18:20.117431 8255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:20.120645 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.110132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52577 (local address 127.0.98.62:46313)
0620 00:18:20.110189 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.110191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.110412 (+ 221us) server_negotiation.cc:407] Connection header received
0620 00:18:20.110903 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.110905 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.110919 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.110999 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:20.112055 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.112942 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.114467 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.114708 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.114794 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.117501 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:20.117502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:20.117503 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:20.117512 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:20.119082 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.119486 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.119487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.119487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.119548 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.119949 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.119950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.119951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.120082 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:20.120229 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.120498 (+ 269us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.120580 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260620 00:18:20.120913 8279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.110085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46313 (local address 127.0.98.3:52577)
0620 00:18:20.110251 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.110292 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.110805 (+ 513us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.111147 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.111150 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.111602 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:20.111926 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.111934 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.113052 (+ 1118us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.113053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.114339 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.114343 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.114818 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.114820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.114963 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.115581 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:20.115591 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:20.117391 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:20.119158 (+ 1767us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:20.119163 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:20.119165 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:20.119381 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:20.119661 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:20.119661 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:20.119662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:20.119821 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:20.120210 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:20.120217 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:20.120367 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.120459 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.120597 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":77,"threads_started":1}
I20260620 00:18:20.121389 8386 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46313
I20260620 00:18:20.121498 8386 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:20.121701 8386 heartbeater.cc:511] Master 127.0.98.62:46313 requested a full tablet report, sending...
I20260620 00:18:20.122118 7903 ts_manager.cc:194] Registered new tserver with Master: 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389)
I20260620 00:18:20.122637 7903 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52577
I20260620 00:18:20.124781 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 8255
I20260620 00:18:20.124869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:20.125083 392 external_mini_cluster.cc:1428] Setting key 122f355b42156035e70a0ca9f6bac60a
I20260620 00:18:20.126134 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:18:20.128955 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.122546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59921 (local address 127.0.98.62:46313)
0620 00:18:20.122585 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.122587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.122689 (+ 102us) server_negotiation.cc:407] Connection header received
0620 00:18:20.122795 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.122796 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.122805 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.122875 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:20.123226 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.123686 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.124512 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.124692 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.124750 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.125980 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:20.125981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:20.125982 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:20.125989 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:20.127811 (+ 1822us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.128131 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.128132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.128157 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.128219 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.128413 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.128414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.128414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.128539 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:20.128611 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.128782 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.128863 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:18:20.129015 8392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.122438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46313 (local address 127.0.98.3:59921)
0620 00:18:20.122620 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.122631 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.122704 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.122862 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.122863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.123072 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:20.123144 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.123146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.123829 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.123830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.124436 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.124438 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.124818 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.124820 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.124902 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.125451 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:20.125457 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:20.125888 (+ 431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:20.127897 (+ 2009us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:20.127898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:20.127898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:20.128058 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:20.128284 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:20.128284 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:20.128285 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:20.128333 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:20.128614 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:20.128615 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:20.128659 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.128739 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.128908 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":75,"threads_started":1}
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:18:20.149214 7953 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:20.143459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46560 (local address 127.0.98.62:46313)
0620 00:18:20.143545 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.143546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.143555 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:20.143581 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.143582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.143591 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.143646 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:20.144331 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.144875 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.145677 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.145899 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.145955 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.149151 (+ 3196us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46560: BlockingRecv error: recv got EOF from 127.0.0.1:46560 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914700, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:20.156648 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.150033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46574 (local address 127.0.98.62:46313)
0620 00:18:20.150115 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.150117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.150207 (+ 90us) server_negotiation.cc:407] Connection header received
0620 00:18:20.150286 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.150287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.150296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.150384 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:20.150926 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.151412 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.152153 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.152358 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.152440 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.154513 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:20.154514 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:20.154514 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:20.154521 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:20.155666 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.155971 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.155971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.155972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.156021 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.156231 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.156232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.156232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.156353 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:20.156437 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.156528 (+ 91us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.156586 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260620 00:18:20.158226 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:18:20.159796 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:20.166945 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.162588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60174 (local address 127.0.98.2:42547)
0620 00:18:20.162766 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.162772 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.162783 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:20.162832 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.162834 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.162872 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.163189 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:20.163378 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.164317 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.165761 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.166505 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.166585 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.166672 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.166789 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.166832 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":82,"threads_started":1}
I20260620 00:18:20.166965 8408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.162205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35389 (local address 127.0.0.1:42672)
0620 00:18:20.162325 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.162370 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.162442 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.162976 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.162979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.162982 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:20.163060 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.163062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.164487 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.164487 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.165845 (+ 1358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.165847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.166717 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.166718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.166781 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.166784 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.166832 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.166867 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"mutex_wait_us":48,"thread_start_us":71,"threads_started":1}
I20260620 00:18:20.167348 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.162368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42672 (local address 127.0.98.3:35389)
0620 00:18:20.162784 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.162789 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.162801 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:18:20.162844 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.162846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.162862 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.162971 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:20.163113 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.164419 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.165947 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.166637 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.166707 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.166775 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.167194 (+ 419us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.167237 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361,"thread_start_us":88,"threads_started":1}
I20260620 00:18:20.166965 8409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.162239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42547 (local address 127.0.0.1:60174)
0620 00:18:20.162389 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.162399 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.162442 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.163246 (+ 804us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.163246 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.163248 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:20.163284 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.163285 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.164433 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.164433 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.165670 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.165673 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.166592 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.166593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.166653 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.166665 (+ 12us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.166768 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.166888 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":82,"thread_start_us":98,"threads_started":1}
I20260620 00:18:20.167999 8411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.162412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41723 (local address 127.0.0.1:54986)
0620 00:18:20.163116 (+ 704us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.163126 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.163181 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.163428 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.163429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.163431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:20.163485 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.163487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.164714 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.164715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.165860 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.165862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.167769 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.167770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.167830 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.167834 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.167881 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.167920 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":71,"threads_started":1}
I20260620 00:18:20.168067 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 33a82c36ad3e4848b9619599059ef389 (DEFAULT_TABLE table=test-table [id=ed4709c4ece449b2928bb002629429d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:20.168272 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.162637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.0.98.1:41723)
0620 00:18:20.162833 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.162837 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.163235 (+ 398us) server_negotiation.cc:407] Connection header received
0620 00:18:20.163276 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.163278 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.163293 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.163358 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:20.163567 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.164629 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.166814 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.167602 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.167667 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.167736 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.168107 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.168173 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":106,"threads_started":1}
I20260620 00:18:20.168267 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 33a82c36ad3e4848b9619599059ef389 (DEFAULT_TABLE table=test-table [id=ed4709c4ece449b2928bb002629429d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:20.168521 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a82c36ad3e4848b9619599059ef389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.168668 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a82c36ad3e4848b9619599059ef389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.171342 8414 tablet_bootstrap.cc:492] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: Bootstrap starting.
I20260620 00:18:20.172073 8414 tablet_bootstrap.cc:654] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.172462 8414 log.cc:824] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:20.172413 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 33a82c36ad3e4848b9619599059ef389 (DEFAULT_TABLE table=test-table [id=ed4709c4ece449b2928bb002629429d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:20.172752 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 33a82c36ad3e4848b9619599059ef389. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.172796 8415 tablet_bootstrap.cc:492] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: Bootstrap starting.
I20260620 00:18:20.173213 8414 tablet_bootstrap.cc:492] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: No bootstrap required, opened a new log
I20260620 00:18:20.173305 8414 ts_tablet_manager.cc:1397] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:18:20.173395 8415 tablet_bootstrap.cc:654] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.173707 8415 log.cc:824] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:20.174422 8415 tablet_bootstrap.cc:492] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: No bootstrap required, opened a new log
I20260620 00:18:20.174491 8415 ts_tablet_manager.cc:1397] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:18:20.175078 8414 raft_consensus.cc:348] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } }
I20260620 00:18:20.175238 8414 raft_consensus.cc:374] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.175283 8414 raft_consensus.cc:729] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7f02ab3881440abe79ebb295000a38, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.175418 8414 consensus_queue.cc:260] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } }
I20260620 00:18:20.175709 8414 ts_tablet_manager.cc:1428] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:18:20.175709 8386 heartbeater.cc:503] Master 127.0.98.62:46313 was elected leader, sending a full tablet report...
I20260620 00:18:20.176694 8415 raft_consensus.cc:348] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } }
I20260620 00:18:20.176873 8415 raft_consensus.cc:374] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.176918 8415 raft_consensus.cc:729] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61f3b0f50bc4e6db51ebe4acf8d721e, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.176846 8419 tablet_bootstrap.cc:492] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: Bootstrap starting.
I20260620 00:18:20.177047 8415 consensus_queue.cc:260] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } }
I20260620 00:18:20.177347 8415 ts_tablet_manager.cc:1428] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:18:20.177392 8246 heartbeater.cc:503] Master 127.0.98.62:46313 was elected leader, sending a full tablet report...
I20260620 00:18:20.177783 8419 tablet_bootstrap.cc:654] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.178148 8419 log.cc:824] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:20.178812 8419 tablet_bootstrap.cc:492] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: No bootstrap required, opened a new log
I20260620 00:18:20.178865 8419 ts_tablet_manager.cc:1397] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:18:20.180313 8419 raft_consensus.cc:348] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } }
I20260620 00:18:20.180485 8419 raft_consensus.cc:374] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.180549 8419 raft_consensus.cc:729] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d87d9c06e364abda3c8cdc8fa885787, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.180665 8419 consensus_queue.cc:260] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } }
I20260620 00:18:20.180917 8419 ts_tablet_manager.cc:1428] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:18:20.180976 8106 heartbeater.cc:503] Master 127.0.98.62:46313 was elected leader, sending a full tablet report...
W20260620 00:18:20.211140 8247 tablet.cc:2369] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:18:20.223686 8420 raft_consensus.cc:482] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:20.223852 8420 raft_consensus.cc:504] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } }
I20260620 00:18:20.224334 8420 leader_election.cc:288] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389), 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723)
I20260620 00:18:20.227385 8410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.224594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43905 (local address 127.0.98.3:35389)
0620 00:18:20.224658 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.224661 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.224670 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:20.224695 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.224696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.224707 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.224762 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:20.224924 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.225436 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.226536 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.227031 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.227086 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.227095 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.227282 (+ 187us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.227312 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:18:20.227389 8139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.224491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35389 (local address 127.0.98.2:43905)
0620 00:18:20.224541 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.224555 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.224606 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.224774 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.224774 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.224776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:20.224847 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.224849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.225522 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.225523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.226432 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.226434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.227109 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.227110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.227202 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.227206 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.227272 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.227315 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 00:18:20.227771 8423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.224626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41723 (local address 127.0.98.2:51899)
0620 00:18:20.224833 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.224853 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.224929 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.225080 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.225081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.225083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:20.225160 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.225162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.225859 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.225860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.226703 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.226706 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.227523 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.227524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.227602 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.227607 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.227649 (+ 42us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.227686 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":118,"threads_started":1}
I20260620 00:18:20.227769 8413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.224723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51899 (local address 127.0.98.1:41723)
0620 00:18:20.224783 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.224785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.224922 (+ 137us) server_negotiation.cc:407] Connection header received
0620 00:18:20.224999 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.225000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.225012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.225061 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:20.225233 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.225746 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.226821 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.227390 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.227458 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.227470 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.227669 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.227706 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:18:20.227747 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a82c36ad3e4848b9619599059ef389" candidate_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7f02ab3881440abe79ebb295000a38" is_pre_election: true
I20260620 00:18:20.227939 8340 raft_consensus.cc:2454] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a61f3b0f50bc4e6db51ebe4acf8d721e in term 0.
I20260620 00:18:20.228046 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a82c36ad3e4848b9619599059ef389" candidate_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d87d9c06e364abda3c8cdc8fa885787" is_pre_election: true
I20260620 00:18:20.228214 8060 raft_consensus.cc:2454] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a61f3b0f50bc4e6db51ebe4acf8d721e in term 0.
I20260620 00:18:20.228232 8134 leader_election.cc:302] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7f02ab3881440abe79ebb295000a38, a61f3b0f50bc4e6db51ebe4acf8d721e; no voters:
I20260620 00:18:20.228353 8420 raft_consensus.cc:2781] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:20.228449 8420 raft_consensus.cc:482] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:20.228487 8420 raft_consensus.cc:3037] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:20.229431 8420 raft_consensus.cc:504] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } }
I20260620 00:18:20.229585 8420 leader_election.cc:288] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [CANDIDATE]: Term 1 election: Requested vote from peers 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389), 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723)
I20260620 00:18:20.229761 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a82c36ad3e4848b9619599059ef389" candidate_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d87d9c06e364abda3c8cdc8fa885787"
I20260620 00:18:20.229761 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "33a82c36ad3e4848b9619599059ef389" candidate_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7f02ab3881440abe79ebb295000a38"
I20260620 00:18:20.229851 8060 raft_consensus.cc:3037] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:20.229902 8340 raft_consensus.cc:3037] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:20.230767 8340 raft_consensus.cc:2454] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a61f3b0f50bc4e6db51ebe4acf8d721e in term 1.
I20260620 00:18:20.230767 8060 raft_consensus.cc:2454] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a61f3b0f50bc4e6db51ebe4acf8d721e in term 1.
I20260620 00:18:20.230997 8134 leader_election.cc:302] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7f02ab3881440abe79ebb295000a38, a61f3b0f50bc4e6db51ebe4acf8d721e; no voters:
I20260620 00:18:20.231101 8420 raft_consensus.cc:2781] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:20.231266 8420 raft_consensus.cc:686] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 LEADER]: Becoming Leader. State: Replica: a61f3b0f50bc4e6db51ebe4acf8d721e, State: Running, Role: LEADER
I20260620 00:18:20.231380 8420 consensus_queue.cc:237] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } }
I20260620 00:18:20.232362 7902 catalog_manager.cc:5495] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e reported cstate change: term changed from 0 to 1, leader changed from <none> to a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } health_report { overall_health: HEALTHY } } }
I20260620 00:18:20.244531 8412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.242012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60184 (local address 127.0.98.2:42547)
0620 00:18:20.242075 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.242077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.242086 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:20.242138 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.242139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.242158 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.242240 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:20.242394 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.242870 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.243674 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.243871 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.243947 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.244130 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:20.244297 (+ 167us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:20.244395 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.244420 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.244441 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:18:20.246431 8340 raft_consensus.cc:1261] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Refusing update from remote peer a61f3b0f50bc4e6db51ebe4acf8d721e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:20.246433 8060 raft_consensus.cc:1261] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Refusing update from remote peer a61f3b0f50bc4e6db51ebe4acf8d721e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:20.246838 8424 consensus_queue.cc:1035] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:20.247014 8420 consensus_queue.cc:1035] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:20.250243 8429 mvcc.cc:205] Tried to move back new op lower bound from 7298722612206432256 to 7298722612148420608. Current Snapshot: MvccSnapshot[applied={T|T < 7298722612206432256}]
I20260620 00:18:20.252110 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:20.252249 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:20.253111 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267: Sending DeleteTablet for 3 replicas of tablet 33a82c36ad3e4848b9619599059ef389
I20260620 00:18:20.253505 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a82c36ad3e4848b9619599059ef389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54986
I20260620 00:18:20.253540 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a82c36ad3e4848b9619599059ef389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42672
I20260620 00:18:20.253806 8437 tablet_replica.cc:330] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: stopping tablet replica
I20260620 00:18:20.253878 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 33a82c36ad3e4848b9619599059ef389 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60174
I20260620 00:18:20.253916 8437 raft_consensus.cc:2229] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:20.254086 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46574:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:18:20.254230 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:20.254354 8437 raft_consensus.cc:2258] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:20.254182 8438 tablet_replica.cc:330] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: stopping tablet replica
I20260620 00:18:20.254570 8438 raft_consensus.cc:2229] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:18:20.254714 8438 raft_consensus.cc:2258] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:20.254853 8437 ts_tablet_manager.cc:1905] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:20.255435 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 7f5815c4e8d3424d84a62cfcb172560c (DEFAULT_TABLE table=test-table [id=f290d3b5897f4d6a9d98004d4819477a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:20.255570 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f5815c4e8d3424d84a62cfcb172560c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.255779 8438 ts_tablet_manager.cc:1905] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:20.256575 8437 ts_tablet_manager.cc:1918] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:20.256654 8437 log.cc:1196] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/33a82c36ad3e4848b9619599059ef389
I20260620 00:18:20.256928 8419 tablet_bootstrap.cc:492] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: Bootstrap starting.
I20260620 00:18:20.256923 8437 ts_tablet_manager.cc:1939] T 33a82c36ad3e4848b9619599059ef389 P 1b7f02ab3881440abe79ebb295000a38: Deleting consensus metadata
I20260620 00:18:20.257652 8419 tablet_bootstrap.cc:654] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.257916 8438 ts_tablet_manager.cc:1918] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:20.258003 8438 log.cc:1196] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/33a82c36ad3e4848b9619599059ef389
I20260620 00:18:20.258148 7890 catalog_manager.cc:4841] TS 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389): tablet 33a82c36ad3e4848b9619599059ef389 (table test-table [id=ed4709c4ece449b2928bb002629429d4]) successfully deleted
I20260620 00:18:20.258216 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 7f5815c4e8d3424d84a62cfcb172560c (DEFAULT_TABLE table=test-table [id=f290d3b5897f4d6a9d98004d4819477a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:20.258275 8438 ts_tablet_manager.cc:1939] T 33a82c36ad3e4848b9619599059ef389 P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting consensus metadata
I20260620 00:18:20.258327 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f5815c4e8d3424d84a62cfcb172560c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.259538 8414 tablet_bootstrap.cc:492] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: Bootstrap starting.
I20260620 00:18:20.259588 7887 catalog_manager.cc:4841] TS a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547): tablet 33a82c36ad3e4848b9619599059ef389 (table test-table [id=ed4709c4ece449b2928bb002629429d4]) successfully deleted
I20260620 00:18:20.259783 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 7f5815c4e8d3424d84a62cfcb172560c (DEFAULT_TABLE table=test-table [id=f290d3b5897f4d6a9d98004d4819477a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:20.259850 8436 tablet_replica.cc:330] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: stopping tablet replica
I20260620 00:18:20.259878 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f5815c4e8d3424d84a62cfcb172560c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.259951 8414 tablet_bootstrap.cc:654] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.259956 8436 raft_consensus.cc:2229] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:20.260071 8436 raft_consensus.cc:2258] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:20.260522 8436 ts_tablet_manager.cc:1905] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:20.261349 8415 tablet_bootstrap.cc:492] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: Bootstrap starting.
I20260620 00:18:20.261718 8415 tablet_bootstrap.cc:654] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.262120 8436 ts_tablet_manager.cc:1918] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:20.262198 8436 log.cc:1196] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/33a82c36ad3e4848b9619599059ef389
I20260620 00:18:20.262380 8415 tablet_bootstrap.cc:492] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: No bootstrap required, opened a new log
I20260620 00:18:20.262430 8415 ts_tablet_manager.cc:1397] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:20.262478 8436 ts_tablet_manager.cc:1939] T 33a82c36ad3e4848b9619599059ef389 P 8d87d9c06e364abda3c8cdc8fa885787: Deleting consensus metadata
I20260620 00:18:20.262573 8415 raft_consensus.cc:348] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.262650 8415 raft_consensus.cc:374] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.262672 8415 raft_consensus.cc:729] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61f3b0f50bc4e6db51ebe4acf8d721e, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.262719 8415 consensus_queue.cc:260] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.262828 8415 ts_tablet_manager.cc:1428] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:20.262878 7890 catalog_manager.cc:4841] TS 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723): tablet 33a82c36ad3e4848b9619599059ef389 (table test-table [id=ed4709c4ece449b2928bb002629429d4]) successfully deleted
I20260620 00:18:20.263021 8414 tablet_bootstrap.cc:492] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: No bootstrap required, opened a new log
I20260620 00:18:20.263067 8414 ts_tablet_manager.cc:1397] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260620 00:18:20.263201 8414 raft_consensus.cc:348] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.263270 8414 raft_consensus.cc:374] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.263294 8414 raft_consensus.cc:729] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7f02ab3881440abe79ebb295000a38, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.263339 8414 consensus_queue.cc:260] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.263432 8414 ts_tablet_manager.cc:1428] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:20.264523 8419 tablet_bootstrap.cc:492] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: No bootstrap required, opened a new log
I20260620 00:18:20.264573 8419 ts_tablet_manager.cc:1397] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.000s
I20260620 00:18:20.264746 8419 raft_consensus.cc:348] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.264820 8419 raft_consensus.cc:374] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.264854 8419 raft_consensus.cc:729] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d87d9c06e364abda3c8cdc8fa885787, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.264902 8419 consensus_queue.cc:260] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.264994 8419 ts_tablet_manager.cc:1428] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260620 00:18:20.277652 8107 tablet.cc:2369] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:18:20.368013 8387 tablet.cc:2369] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:18:20.614089 8424 raft_consensus.cc:482] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:20.614239 8424 raft_consensus.cc:504] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.614427 8424 leader_election.cc:288] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723), 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389)
I20260620 00:18:20.614732 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5815c4e8d3424d84a62cfcb172560c" candidate_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d87d9c06e364abda3c8cdc8fa885787" is_pre_election: true
I20260620 00:18:20.614759 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5815c4e8d3424d84a62cfcb172560c" candidate_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7f02ab3881440abe79ebb295000a38" is_pre_election: true
I20260620 00:18:20.614856 8060 raft_consensus.cc:2454] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a61f3b0f50bc4e6db51ebe4acf8d721e in term 0.
I20260620 00:18:20.614871 8340 raft_consensus.cc:2454] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a61f3b0f50bc4e6db51ebe4acf8d721e in term 0.
I20260620 00:18:20.615025 8134 leader_election.cc:302] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7f02ab3881440abe79ebb295000a38, a61f3b0f50bc4e6db51ebe4acf8d721e; no voters:
I20260620 00:18:20.615130 8424 raft_consensus.cc:2781] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:20.615171 8424 raft_consensus.cc:482] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:20.615198 8424 raft_consensus.cc:3037] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:20.615957 8424 raft_consensus.cc:504] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.616103 8424 leader_election.cc:288] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [CANDIDATE]: Term 1 election: Requested vote from peers 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723), 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389)
I20260620 00:18:20.616315 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5815c4e8d3424d84a62cfcb172560c" candidate_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7f02ab3881440abe79ebb295000a38"
I20260620 00:18:20.616285 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f5815c4e8d3424d84a62cfcb172560c" candidate_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d87d9c06e364abda3c8cdc8fa885787"
I20260620 00:18:20.616391 8340 raft_consensus.cc:3037] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:20.616391 8060 raft_consensus.cc:3037] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:20.617110 8340 raft_consensus.cc:2454] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a61f3b0f50bc4e6db51ebe4acf8d721e in term 1.
I20260620 00:18:20.617110 8060 raft_consensus.cc:2454] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a61f3b0f50bc4e6db51ebe4acf8d721e in term 1.
I20260620 00:18:20.617309 8134 leader_election.cc:302] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7f02ab3881440abe79ebb295000a38, a61f3b0f50bc4e6db51ebe4acf8d721e; no voters:
I20260620 00:18:20.617411 8424 raft_consensus.cc:2781] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:20.617527 8424 raft_consensus.cc:686] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 LEADER]: Becoming Leader. State: Replica: a61f3b0f50bc4e6db51ebe4acf8d721e, State: Running, Role: LEADER
I20260620 00:18:20.617581 8424 consensus_queue.cc:237] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.618194 7902 catalog_manager.cc:5495] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e reported cstate change: term changed from 0 to 1, leader changed from <none> to a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2). New cstate: current_term: 1 leader_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } health_report { overall_health: UNKNOWN } } }
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914699, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:20.728334 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.720948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46313 (local address 127.0.0.1:46582)
0620 00:18:20.721127 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.721139 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.721261 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.721462 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.721463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.721778 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:20.721831 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.721833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.722559 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.722561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.723110 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.723111 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.723471 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.723472 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.723510 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.724071 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:20.724076 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:20.725483 (+ 1407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:20.727184 (+ 1701us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:20.727191 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:20.727193 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:20.727386 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:20.727587 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:20.727588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:20.727589 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:20.727655 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:20.727964 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:20.727969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:20.728043 (+ 74us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.728115 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.728257 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":84,"threads_started":1}
I20260620 00:18:20.728447 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.721026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46582 (local address 127.0.98.62:46313)
0620 00:18:20.721212 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.721214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.721223 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:20.721359 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.721360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.721369 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.721434 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:20.721915 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.722477 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.723184 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.723387 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.723442 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.725570 (+ 2128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:20.725572 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:20.725572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:20.725578 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:20.727107 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.727454 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.727455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.727455 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.727509 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.727721 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.727722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.727723 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.727883 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:20.727966 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.728187 (+ 221us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.728317 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":164,"thread_start_us":128,"threads_started":1}
I20260620 00:18:20.730474 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46582:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260620 00:18:20.730702 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:20.732291 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 227a29ed427f410482636028d7cefca3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c2115fb4d5d4f168c82f26899d7edef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:18:20.732317 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 227a29ed427f410482636028d7cefca3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c2115fb4d5d4f168c82f26899d7edef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:18:20.732331 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 227a29ed427f410482636028d7cefca3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=6c2115fb4d5d4f168c82f26899d7edef]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260620 00:18:20.732470 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227a29ed427f410482636028d7cefca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.732473 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227a29ed427f410482636028d7cefca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.732487 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 227a29ed427f410482636028d7cefca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.734005 8419 tablet_bootstrap.cc:492] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: Bootstrap starting.
I20260620 00:18:20.734007 8415 tablet_bootstrap.cc:492] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: Bootstrap starting.
I20260620 00:18:20.734362 8415 tablet_bootstrap.cc:654] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.734366 8419 tablet_bootstrap.cc:654] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.734581 8414 tablet_bootstrap.cc:492] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: Bootstrap starting.
I20260620 00:18:20.734968 8414 tablet_bootstrap.cc:654] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.735005 8419 tablet_bootstrap.cc:492] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: No bootstrap required, opened a new log
I20260620 00:18:20.735038 8419 ts_tablet_manager.cc:1397] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:20.735152 8419 raft_consensus.cc:348] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.735260 8419 raft_consensus.cc:374] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.735283 8419 raft_consensus.cc:729] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d87d9c06e364abda3c8cdc8fa885787, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.735332 8419 consensus_queue.cc:260] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.735435 8415 tablet_bootstrap.cc:492] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: No bootstrap required, opened a new log
I20260620 00:18:20.735481 8415 ts_tablet_manager.cc:1397] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:20.735484 8419 ts_tablet_manager.cc:1428] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:20.735517 8422 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.735579 8415 raft_consensus.cc:348] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.735643 8415 raft_consensus.cc:374] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.735620 8422 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:18:20.735666 8415 raft_consensus.cc:729] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61f3b0f50bc4e6db51ebe4acf8d721e, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.735711 8415 consensus_queue.cc:260] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.735818 8415 ts_tablet_manager.cc:1428] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:20.735816 8424 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.735944 8424 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:18:20.736325 8414 tablet_bootstrap.cc:492] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: No bootstrap required, opened a new log
I20260620 00:18:20.736380 8414 ts_tablet_manager.cc:1397] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:18:20.736512 8414 raft_consensus.cc:348] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.736588 8414 raft_consensus.cc:374] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.736624 8414 raft_consensus.cc:729] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7f02ab3881440abe79ebb295000a38, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.736678 8414 consensus_queue.cc:260] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.736766 8414 ts_tablet_manager.cc:1428] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:20.736837 8418 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.736924 8418 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:18:20.858414 8422 raft_consensus.cc:482] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:20.858526 8422 raft_consensus.cc:504] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.858989 8422 leader_election.cc:288] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547), 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389)
I20260620 00:18:20.862020 8456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.859388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49827 (local address 127.0.98.3:35389)
0620 00:18:20.859499 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.859501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.859508 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:18:20.859531 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.859531 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.859541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.859621 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:20.859777 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.860271 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.861269 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.861654 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.861740 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.861754 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.861898 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.861939 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
I20260620 00:18:20.862040 8455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.859267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35389 (local address 127.0.98.1:49827)
0620 00:18:20.859362 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.859373 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.859427 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.859625 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.859626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.859628 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:20.859701 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.859702 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.860369 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.860370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.861171 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.861173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.861741 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.861742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.861818 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.861821 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.861888 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.861929 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"mutex_wait_us":2,"thread_start_us":41,"threads_started":1}
I20260620 00:18:20.862056 8453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.859180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42547 (local address 127.0.98.1:56305)
0620 00:18:20.859311 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.859325 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.859388 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.859588 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.859589 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.859605 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:20.859673 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.859675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.860400 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.860401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.861180 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.861183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.861756 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.861757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.861841 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.861843 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.861911 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.861956 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":28,"thread_start_us":70,"threads_started":1}
I20260620 00:18:20.862035 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.859242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56305 (local address 127.0.98.2:42547)
0620 00:18:20.859367 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.859369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.859382 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:18:20.859495 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.859496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.859506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.859568 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:20.859750 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.860318 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.861276 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.861672 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.861756 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.861767 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.861916 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.861957 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":62,"threads_started":1}
I20260620 00:18:20.862293 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227a29ed427f410482636028d7cefca3" candidate_uuid: "8d87d9c06e364abda3c8cdc8fa885787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7f02ab3881440abe79ebb295000a38" is_pre_election: true
I20260620 00:18:20.862373 8340 raft_consensus.cc:2454] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d87d9c06e364abda3c8cdc8fa885787 in term 0.
I20260620 00:18:20.862448 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227a29ed427f410482636028d7cefca3" candidate_uuid: "8d87d9c06e364abda3c8cdc8fa885787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" is_pre_election: true
I20260620 00:18:20.862525 7994 leader_election.cc:302] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7f02ab3881440abe79ebb295000a38, 8d87d9c06e364abda3c8cdc8fa885787; no voters:
I20260620 00:18:20.862573 8200 raft_consensus.cc:2454] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d87d9c06e364abda3c8cdc8fa885787 in term 0.
I20260620 00:18:20.862658 8422 raft_consensus.cc:2781] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:20.862725 8422 raft_consensus.cc:482] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:20.862751 8422 raft_consensus.cc:3037] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:20.863536 8422 raft_consensus.cc:504] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.863675 8422 leader_election.cc:288] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [CANDIDATE]: Term 1 election: Requested vote from peers a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547), 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389)
I20260620 00:18:20.863799 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227a29ed427f410482636028d7cefca3" candidate_uuid: "8d87d9c06e364abda3c8cdc8fa885787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e"
I20260620 00:18:20.863798 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "227a29ed427f410482636028d7cefca3" candidate_uuid: "8d87d9c06e364abda3c8cdc8fa885787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7f02ab3881440abe79ebb295000a38"
I20260620 00:18:20.863867 8200 raft_consensus.cc:3037] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:20.863888 8340 raft_consensus.cc:3037] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:20.864591 8340 raft_consensus.cc:2454] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d87d9c06e364abda3c8cdc8fa885787 in term 1.
I20260620 00:18:20.864615 8200 raft_consensus.cc:2454] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d87d9c06e364abda3c8cdc8fa885787 in term 1.
I20260620 00:18:20.864758 7994 leader_election.cc:302] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7f02ab3881440abe79ebb295000a38, 8d87d9c06e364abda3c8cdc8fa885787; no voters:
I20260620 00:18:20.864852 8422 raft_consensus.cc:2781] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:20.864979 8422 raft_consensus.cc:686] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 LEADER]: Becoming Leader. State: Replica: 8d87d9c06e364abda3c8cdc8fa885787, State: Running, Role: LEADER
I20260620 00:18:20.865059 8422 consensus_queue.cc:237] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.865342 8457 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: TxnStatusTablet state changed. Reason: New leader 8d87d9c06e364abda3c8cdc8fa885787. Latest consensus state: current_term: 1 leader_uuid: "8d87d9c06e364abda3c8cdc8fa885787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.865439 8457 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:18:20.865615 8459 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260620 00:18:20.865746 8459 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260620 00:18:20.865844 7903 catalog_manager.cc:5495] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8d87d9c06e364abda3c8cdc8fa885787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } health_report { overall_health: UNKNOWN } } }
I20260620 00:18:20.884611 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.881742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55000 (local address 127.0.98.1:41723)
0620 00:18:20.881920 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.881922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.881931 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:20.881974 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.881975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.881995 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.882088 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:20.882239 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.882903 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.883682 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.883910 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.883986 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.884214 (+ 228us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:20.884336 (+ 122us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:20.884417 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.884477 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.884500 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":93,"threads_started":1}
I20260620 00:18:20.884603 8448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.881697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41723 (local address 127.0.0.1:55000)
0620 00:18:20.881775 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.881792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.881889 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.882091 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.882092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.882094 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:18:20.882156 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.882159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.882983 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.882985 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.883577 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.883579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.883985 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.883986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.884045 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.884066 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:18:20.884419 (+ 353us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:18:20.884429 (+ 10us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.884474 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.884530 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260620 00:18:20.885921 8340 raft_consensus.cc:1261] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Refusing update from remote peer 8d87d9c06e364abda3c8cdc8fa885787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:20.886217 8200 raft_consensus.cc:1261] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Refusing update from remote peer 8d87d9c06e364abda3c8cdc8fa885787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:20.886262 8457 consensus_queue.cc:1035] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:20.886488 8422 consensus_queue.cc:1035] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:20.886935 8418 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: TxnStatusTablet state changed. Reason: New leader 8d87d9c06e364abda3c8cdc8fa885787. Latest consensus state: current_term: 1 leader_uuid: "8d87d9c06e364abda3c8cdc8fa885787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.887048 8418 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:18:20.887308 8424 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: TxnStatusTablet state changed. Reason: New leader 8d87d9c06e364abda3c8cdc8fa885787. Latest consensus state: current_term: 1 leader_uuid: "8d87d9c06e364abda3c8cdc8fa885787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.887396 8424 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:18:20.887861 8422 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d87d9c06e364abda3c8cdc8fa885787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.887866 8418 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d87d9c06e364abda3c8cdc8fa885787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.887939 8418 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P 1b7f02ab3881440abe79ebb295000a38: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:18:20.887938 8422 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:18:20.888386 8463 mvcc.cc:205] Tried to move back new op lower bound from 7298722614826688512 to 7298722614743855104. Current Snapshot: MvccSnapshot[applied={T|T < 7298722614826688512}]
I20260620 00:18:20.888568 8457 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8d87d9c06e364abda3c8cdc8fa885787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.888639 8457 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P 8d87d9c06e364abda3c8cdc8fa885787: This TxnStatusTablet replica's current role is: LEADER
I20260620 00:18:20.888877 8424 tablet_replica.cc:439] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8d87d9c06e364abda3c8cdc8fa885787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } } }
I20260620 00:18:20.888918 8466 mvcc.cc:205] Tried to move back new op lower bound from 7298722614826688512 to 7298722614743855104. Current Snapshot: MvccSnapshot[applied={T|T < 7298722614826688512}]
I20260620 00:18:20.888964 8424 tablet_replica.cc:442] T 227a29ed427f410482636028d7cefca3 P a61f3b0f50bc4e6db51ebe4acf8d721e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260620 00:18:20.899680 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.893901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57629 (local address 127.0.98.1:41723)
0620 00:18:20.893967 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.893970 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.896415 (+ 2445us) server_negotiation.cc:407] Connection header received
0620 00:18:20.896456 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.896457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.896469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.896532 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:20.896697 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.897233 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.898696 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.899061 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.899170 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.899365 (+ 195us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:20.899405 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:20.899463 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.899556 (+ 93us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.899582 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:18:20.899992 8473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.893859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41723 (local address 127.0.98.2:57629)
0620 00:18:20.896295 (+ 2436us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.896310 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.896402 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.896549 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.896550 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.896552 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:18:20.896610 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.896612 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.897347 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.897349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.898567 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.898572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.899157 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.899158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.899232 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.899260 (+ 28us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:18:20.899482 (+ 222us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:18:20.899484 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.899542 (+ 58us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.899621 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2376,"thread_start_us":88,"threads_started":1}
I20260620 00:18:20.903129 8340 raft_consensus.cc:1261] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Refusing update from remote peer a61f3b0f50bc4e6db51ebe4acf8d721e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:20.903204 8060 raft_consensus.cc:1261] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Refusing update from remote peer a61f3b0f50bc4e6db51ebe4acf8d721e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:20.903462 8424 consensus_queue.cc:1035] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:20.904130 8424 consensus_queue.cc:1035] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:20.920673 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.917335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39223 (local address 127.0.98.2:42547)
0620 00:18:20.917379 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.917381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.917529 (+ 148us) server_negotiation.cc:407] Connection header received
0620 00:18:20.917596 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.917598 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.917612 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.917664 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:20.917859 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.918596 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.919682 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.920021 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.920100 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.920424 (+ 324us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:20.920464 (+ 40us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:20.920527 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.920563 (+ 36us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.920588 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:18:20.920672 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.917273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42547 (local address 127.0.98.1:39223)
0620 00:18:20.917434 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:20.917445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:20.917543 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:20.917690 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:20.917692 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:20.917694 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:18:20.917778 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.917780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.918805 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.918806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.919597 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:20.919600 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.920188 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.920189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:20.920283 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.920323 (+ 40us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:18:20.920526 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:18:20.920528 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:18:20.920559 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 00:18:20.920596 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260620 00:18:20.929351 7903 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:20.929466 7903 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46574:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:20.930140 7903 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267: Sending DeleteTablet for 3 replicas of tablet 7f5815c4e8d3424d84a62cfcb172560c
I20260620 00:18:20.930434 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f5815c4e8d3424d84a62cfcb172560c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60174
I20260620 00:18:20.930423 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f5815c4e8d3424d84a62cfcb172560c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54986
I20260620 00:18:20.930437 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f5815c4e8d3424d84a62cfcb172560c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:20 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42672
I20260620 00:18:20.930697 8480 tablet_replica.cc:330] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: stopping tablet replica
I20260620 00:18:20.930764 8480 raft_consensus.cc:2229] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:20.930765 8481 tablet_replica.cc:330] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: stopping tablet replica
I20260620 00:18:20.930820 8481 raft_consensus.cc:2229] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:18:20.930825 8480 raft_consensus.cc:2258] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:20.930889 8481 raft_consensus.cc:2258] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:20.931025 8482 tablet_replica.cc:330] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: stopping tablet replica
I20260620 00:18:20.931092 8482 raft_consensus.cc:2229] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:20.931139 8482 raft_consensus.cc:2258] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:20.931243 8481 ts_tablet_manager.cc:1905] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:20.931408 8482 ts_tablet_manager.cc:1905] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:20.933079 8481 ts_tablet_manager.cc:1918] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:18:20.933168 8481 log.cc:1196] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/7f5815c4e8d3424d84a62cfcb172560c
I20260620 00:18:20.933377 8482 ts_tablet_manager.cc:1918] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:18:20.933456 8482 log.cc:1196] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/7f5815c4e8d3424d84a62cfcb172560c
I20260620 00:18:20.933463 8481 ts_tablet_manager.cc:1939] T 7f5815c4e8d3424d84a62cfcb172560c P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting consensus metadata
I20260620 00:18:20.933708 8482 ts_tablet_manager.cc:1939] T 7f5815c4e8d3424d84a62cfcb172560c P 8d87d9c06e364abda3c8cdc8fa885787: Deleting consensus metadata
I20260620 00:18:20.933811 7887 catalog_manager.cc:4841] TS a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547): tablet 7f5815c4e8d3424d84a62cfcb172560c (table test-table [id=f290d3b5897f4d6a9d98004d4819477a]) successfully deleted
I20260620 00:18:20.934103 7890 catalog_manager.cc:4841] TS 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723): tablet 7f5815c4e8d3424d84a62cfcb172560c (table test-table [id=f290d3b5897f4d6a9d98004d4819477a]) successfully deleted
I20260620 00:18:20.935320 8480 ts_tablet_manager.cc:1905] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:20.937009 8480 ts_tablet_manager.cc:1918] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:18:20.937079 8480 log.cc:1196] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/7f5815c4e8d3424d84a62cfcb172560c
I20260620 00:18:20.937284 8480 ts_tablet_manager.cc:1939] T 7f5815c4e8d3424d84a62cfcb172560c P 1b7f02ab3881440abe79ebb295000a38: Deleting consensus metadata
I20260620 00:18:20.937661 7890 catalog_manager.cc:4841] TS 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389): tablet 7f5815c4e8d3424d84a62cfcb172560c (table test-table [id=f290d3b5897f4d6a9d98004d4819477a]) successfully deleted
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:18:20.961567 8449 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:20.955173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46594 (local address 127.0.98.62:46313)
0620 00:18:20.955245 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.955248 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.955354 (+ 106us) server_negotiation.cc:407] Connection header received
0620 00:18:20.955415 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.955417 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.955429 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.955509 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:20.956186 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.956996 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.957820 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.958126 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.958213 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.961499 (+ 3286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46594: BlockingRecv error: recv got EOF from 127.0.0.1:46594 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 00:18:20 dist-test-slave-r81b krb5kdc[7858](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260620 00:18:20.972648 8449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:20.964608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46608 (local address 127.0.98.62:46313)
0620 00:18:20.964653 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:20.964655 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:20.964741 (+ 86us) server_negotiation.cc:407] Connection header received
0620 00:18:20.964805 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:20.964806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:20.964817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:20.964918 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:20.965587 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.966084 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.967121 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:20.967298 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:20.967347 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:20.969945 (+ 2598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:20.969946 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:20.969947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:20.969953 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:20.971405 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.971839 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.971840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.971840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.971891 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:20.972098 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:20.972099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:20.972099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:20.972276 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:20.972359 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:20.972516 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 00:18:20.972581 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:18:20.974617 7903 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:18:20.974812 7903 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:20.976285 8040 tablet_service.cc:1467] Processing CreateTablet for tablet 59fe74b04f6c4fb78ff3bafcf3655657 (DEFAULT_TABLE table=test-table [id=7554c090e6fc4a2baa9d5c2f848ff7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:20.976303 8180 tablet_service.cc:1467] Processing CreateTablet for tablet 59fe74b04f6c4fb78ff3bafcf3655657 (DEFAULT_TABLE table=test-table [id=7554c090e6fc4a2baa9d5c2f848ff7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:20.976368 8320 tablet_service.cc:1467] Processing CreateTablet for tablet 59fe74b04f6c4fb78ff3bafcf3655657 (DEFAULT_TABLE table=test-table [id=7554c090e6fc4a2baa9d5c2f848ff7ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:20.976444 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fe74b04f6c4fb78ff3bafcf3655657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.976454 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fe74b04f6c4fb78ff3bafcf3655657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.976472 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59fe74b04f6c4fb78ff3bafcf3655657. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:20.977798 8415 tablet_bootstrap.cc:492] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: Bootstrap starting.
I20260620 00:18:20.977798 8414 tablet_bootstrap.cc:492] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: Bootstrap starting.
I20260620 00:18:20.978132 8415 tablet_bootstrap.cc:654] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.978210 8414 tablet_bootstrap.cc:654] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.978402 8419 tablet_bootstrap.cc:492] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: Bootstrap starting.
I20260620 00:18:20.978829 8419 tablet_bootstrap.cc:654] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:20.978951 8414 tablet_bootstrap.cc:492] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: No bootstrap required, opened a new log
I20260620 00:18:20.978997 8414 ts_tablet_manager.cc:1397] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 00:18:20.979206 8414 raft_consensus.cc:348] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.979287 8414 raft_consensus.cc:374] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.979311 8414 raft_consensus.cc:729] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7f02ab3881440abe79ebb295000a38, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.979352 8414 consensus_queue.cc:260] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.979470 8414 ts_tablet_manager.cc:1428] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:20.979753 8415 tablet_bootstrap.cc:492] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: No bootstrap required, opened a new log
I20260620 00:18:20.979808 8415 ts_tablet_manager.cc:1397] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:18:20.979914 8415 raft_consensus.cc:348] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.979979 8415 raft_consensus.cc:374] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.980000 8415 raft_consensus.cc:729] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61f3b0f50bc4e6db51ebe4acf8d721e, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.980043 8415 consensus_queue.cc:260] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.980088 8419 tablet_bootstrap.cc:492] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: No bootstrap required, opened a new log
I20260620 00:18:20.980157 8419 ts_tablet_manager.cc:1397] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:18:20.980162 8415 ts_tablet_manager.cc:1428] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:20.980298 8419 raft_consensus.cc:348] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.980371 8419 raft_consensus.cc:374] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:20.980402 8419 raft_consensus.cc:729] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d87d9c06e364abda3c8cdc8fa885787, State: Initialized, Role: FOLLOWER
I20260620 00:18:20.980456 8419 consensus_queue.cc:260] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:20.980566 8419 ts_tablet_manager.cc:1428] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:21.173167 8457 raft_consensus.cc:482] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:21.173280 8457 raft_consensus.cc:504] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.173468 8457 leader_election.cc:288] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547), 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389)
I20260620 00:18:21.173754 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fe74b04f6c4fb78ff3bafcf3655657" candidate_uuid: "8d87d9c06e364abda3c8cdc8fa885787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7f02ab3881440abe79ebb295000a38" is_pre_election: true
I20260620 00:18:21.173754 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fe74b04f6c4fb78ff3bafcf3655657" candidate_uuid: "8d87d9c06e364abda3c8cdc8fa885787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" is_pre_election: true
I20260620 00:18:21.173867 8200 raft_consensus.cc:2454] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d87d9c06e364abda3c8cdc8fa885787 in term 0.
I20260620 00:18:21.173867 8340 raft_consensus.cc:2454] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d87d9c06e364abda3c8cdc8fa885787 in term 0.
I20260620 00:18:21.174046 7991 leader_election.cc:302] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d87d9c06e364abda3c8cdc8fa885787, a61f3b0f50bc4e6db51ebe4acf8d721e; no voters:
I20260620 00:18:21.174161 8457 raft_consensus.cc:2781] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:21.174206 8457 raft_consensus.cc:482] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:21.174232 8457 raft_consensus.cc:3037] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:21.174948 8457 raft_consensus.cc:504] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.175098 8457 leader_election.cc:288] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [CANDIDATE]: Term 1 election: Requested vote from peers a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547), 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389)
I20260620 00:18:21.175275 8340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fe74b04f6c4fb78ff3bafcf3655657" candidate_uuid: "8d87d9c06e364abda3c8cdc8fa885787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1b7f02ab3881440abe79ebb295000a38"
I20260620 00:18:21.175275 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59fe74b04f6c4fb78ff3bafcf3655657" candidate_uuid: "8d87d9c06e364abda3c8cdc8fa885787" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e"
I20260620 00:18:21.175364 8200 raft_consensus.cc:3037] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:21.175369 8340 raft_consensus.cc:3037] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:21.176173 8340 raft_consensus.cc:2454] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d87d9c06e364abda3c8cdc8fa885787 in term 1.
I20260620 00:18:21.176169 8200 raft_consensus.cc:2454] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d87d9c06e364abda3c8cdc8fa885787 in term 1.
I20260620 00:18:21.176369 7994 leader_election.cc:302] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7f02ab3881440abe79ebb295000a38, 8d87d9c06e364abda3c8cdc8fa885787; no voters:
I20260620 00:18:21.176496 8457 raft_consensus.cc:2781] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:21.176568 8457 raft_consensus.cc:686] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 LEADER]: Becoming Leader. State: Replica: 8d87d9c06e364abda3c8cdc8fa885787, State: Running, Role: LEADER
I20260620 00:18:21.176620 8457 consensus_queue.cc:237] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.177305 7902 catalog_manager.cc:5495] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "8d87d9c06e364abda3c8cdc8fa885787" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } health_report { overall_health: UNKNOWN } } }
I20260620 00:18:21.209100 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.206894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55014 (local address 127.0.98.1:41723)
0620 00:18:21.206989 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:21.206991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:21.206998 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:18:21.207024 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:21.207024 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:21.207034 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:21.207084 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:21.207211 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.207690 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.208522 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.208717 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.208769 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.208893 (+ 124us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:21.208923 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:21.208960 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:21.209019 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 00:18:21.209037 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 00:18:21.210012 8200 raft_consensus.cc:1261] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Refusing update from remote peer 8d87d9c06e364abda3c8cdc8fa885787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:21.210011 8340 raft_consensus.cc:1261] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Refusing update from remote peer 8d87d9c06e364abda3c8cdc8fa885787: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:21.210283 8422 consensus_queue.cc:1035] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:21.210451 8457 consensus_queue.cc:1035] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:21.213577 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:21.213677 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:21.214291 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267: Sending DeleteTablet for 3 replicas of tablet 59fe74b04f6c4fb78ff3bafcf3655657
I20260620 00:18:21.214567 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet 59fe74b04f6c4fb78ff3bafcf3655657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60174
I20260620 00:18:21.214619 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet 59fe74b04f6c4fb78ff3bafcf3655657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54986
I20260620 00:18:21.214679 8481 tablet_replica.cc:330] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: stopping tablet replica
I20260620 00:18:21.214728 8481 raft_consensus.cc:2229] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:21.214728 8482 tablet_replica.cc:330] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: stopping tablet replica
I20260620 00:18:21.214735 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet 59fe74b04f6c4fb78ff3bafcf3655657 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42672
I20260620 00:18:21.214773 8481 raft_consensus.cc:2258] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:21.214783 8482 raft_consensus.cc:2229] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:18:21.214812 8480 tablet_replica.cc:330] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: stopping tablet replica
I20260620 00:18:21.214866 8480 raft_consensus.cc:2229] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:21.214877 8482 raft_consensus.cc:2258] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:21.214902 8480 raft_consensus.cc:2258] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:21.215183 8482 ts_tablet_manager.cc:1905] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:21.215122 7902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:18:21.215260 7902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:21.215327 8480 ts_tablet_manager.cc:1905] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:21.216697 8320 tablet_service.cc:1467] Processing CreateTablet for tablet a896c1e1be5042e395f369da517bf3f4 (DEFAULT_TABLE table=test-table [id=2c5e038429a843deb6f957edf4cf19c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:21.216888 8482 ts_tablet_manager.cc:1918] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:21.216882 8320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a896c1e1be5042e395f369da517bf3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:21.216938 8482 log.cc:1196] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/59fe74b04f6c4fb78ff3bafcf3655657
I20260620 00:18:21.217146 8482 ts_tablet_manager.cc:1939] T 59fe74b04f6c4fb78ff3bafcf3655657 P 8d87d9c06e364abda3c8cdc8fa885787: Deleting consensus metadata
I20260620 00:18:21.217751 7890 catalog_manager.cc:4841] TS 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723): tablet 59fe74b04f6c4fb78ff3bafcf3655657 (table test-table [id=7554c090e6fc4a2baa9d5c2f848ff7ca]) successfully deleted
I20260620 00:18:21.218137 8414 tablet_bootstrap.cc:492] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: Bootstrap starting.
I20260620 00:18:21.218192 8040 tablet_service.cc:1467] Processing CreateTablet for tablet a896c1e1be5042e395f369da517bf3f4 (DEFAULT_TABLE table=test-table [id=2c5e038429a843deb6f957edf4cf19c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:21.218282 8040 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a896c1e1be5042e395f369da517bf3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:21.218532 8414 tablet_bootstrap.cc:654] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:21.219159 8414 tablet_bootstrap.cc:492] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: No bootstrap required, opened a new log
I20260620 00:18:21.219197 8414 ts_tablet_manager.cc:1397] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260620 00:18:21.219331 8414 raft_consensus.cc:348] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.219398 8414 raft_consensus.cc:374] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:21.219437 8414 raft_consensus.cc:729] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b7f02ab3881440abe79ebb295000a38, State: Initialized, Role: FOLLOWER
I20260620 00:18:21.219488 8414 consensus_queue.cc:260] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.219570 8419 tablet_bootstrap.cc:492] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787: Bootstrap starting.
I20260620 00:18:21.219594 8414 ts_tablet_manager.cc:1428] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:21.219869 8419 tablet_bootstrap.cc:654] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:21.220127 8180 tablet_service.cc:1467] Processing CreateTablet for tablet a896c1e1be5042e395f369da517bf3f4 (DEFAULT_TABLE table=test-table [id=2c5e038429a843deb6f957edf4cf19c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:21.220270 8180 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a896c1e1be5042e395f369da517bf3f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:21.221280 8415 tablet_bootstrap.cc:492] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: Bootstrap starting.
I20260620 00:18:21.216886 8480 ts_tablet_manager.cc:1918] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:21.221501 8480 log.cc:1196] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/59fe74b04f6c4fb78ff3bafcf3655657
I20260620 00:18:21.221647 8415 tablet_bootstrap.cc:654] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:21.221737 8480 ts_tablet_manager.cc:1939] T 59fe74b04f6c4fb78ff3bafcf3655657 P 1b7f02ab3881440abe79ebb295000a38: Deleting consensus metadata
I20260620 00:18:21.222061 7890 catalog_manager.cc:4841] TS 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389): tablet 59fe74b04f6c4fb78ff3bafcf3655657 (table test-table [id=7554c090e6fc4a2baa9d5c2f848ff7ca]) successfully deleted
I20260620 00:18:21.222170 8481 ts_tablet_manager.cc:1905] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:21.222409 8415 tablet_bootstrap.cc:492] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: No bootstrap required, opened a new log
I20260620 00:18:21.222448 8415 ts_tablet_manager.cc:1397] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:21.222548 8415 raft_consensus.cc:348] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.222608 8415 raft_consensus.cc:374] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:21.222627 8415 raft_consensus.cc:729] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a61f3b0f50bc4e6db51ebe4acf8d721e, State: Initialized, Role: FOLLOWER
I20260620 00:18:21.222676 8415 consensus_queue.cc:260] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.222762 8415 ts_tablet_manager.cc:1428] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:21.222962 8419 tablet_bootstrap.cc:492] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787: No bootstrap required, opened a new log
I20260620 00:18:21.223001 8419 ts_tablet_manager.cc:1397] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260620 00:18:21.223107 8419 raft_consensus.cc:348] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.223177 8419 raft_consensus.cc:374] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:21.223199 8419 raft_consensus.cc:729] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d87d9c06e364abda3c8cdc8fa885787, State: Initialized, Role: FOLLOWER
I20260620 00:18:21.223241 8419 consensus_queue.cc:260] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.223316 8419 ts_tablet_manager.cc:1428] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:21.223628 8481 ts_tablet_manager.cc:1918] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:21.223665 8481 log.cc:1196] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/59fe74b04f6c4fb78ff3bafcf3655657
I20260620 00:18:21.223812 8481 ts_tablet_manager.cc:1939] T 59fe74b04f6c4fb78ff3bafcf3655657 P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting consensus metadata
I20260620 00:18:21.224300 7887 catalog_manager.cc:4841] TS a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547): tablet 59fe74b04f6c4fb78ff3bafcf3655657 (table test-table [id=7554c090e6fc4a2baa9d5c2f848ff7ca]) successfully deleted
I20260620 00:18:21.346068 8418 raft_consensus.cc:482] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:21.346237 8418 raft_consensus.cc:504] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.346608 8418 leader_election.cc:288] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723), a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547)
I20260620 00:18:21.349820 8454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.346912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47807 (local address 127.0.98.2:42547)
0620 00:18:21.346961 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:21.346963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:21.347006 (+ 43us) server_negotiation.cc:407] Connection header received
0620 00:18:21.347123 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:21.347124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:21.347134 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:21.347234 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:21.347432 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.348021 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.349098 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.349451 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.349506 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.349515 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:21.349704 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 00:18:21.349744 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:18:21.349830 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.346890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57325 (local address 127.0.98.1:41723)
0620 00:18:21.346952 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:21.346955 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:21.347058 (+ 103us) server_negotiation.cc:407] Connection header received
0620 00:18:21.347140 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:21.347141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:21.347149 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:21.347232 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:21.347407 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.347921 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.349080 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.349468 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.349558 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.349568 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:21.349729 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 00:18:21.349770 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:18:21.349841 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.346804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42547 (local address 127.0.98.3:47807)
0620 00:18:21.346947 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:21.346960 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:21.347028 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:21.347238 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:21.347238 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:21.347240 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:21.347347 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:21.347349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.348111 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.348112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:21.348992 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:21.348995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.349550 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.349551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:21.349617 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.349622 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:18:21.349674 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:18:21.349739 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":21,"thread_start_us":67,"threads_started":1}
I20260620 00:18:21.349853 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.346804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41723 (local address 127.0.98.3:57325)
0620 00:18:21.346968 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:21.346979 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:21.347064 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:21.347231 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:21.347231 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:21.347234 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:21.347338 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:21.347340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.348010 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.348012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:21.348986 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:21.348988 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.349576 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.349578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:21.349640 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.349644 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:21.349714 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 00:18:21.349751 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":58,"thread_start_us":65,"threads_started":1}
I20260620 00:18:21.350066 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a896c1e1be5042e395f369da517bf3f4" candidate_uuid: "1b7f02ab3881440abe79ebb295000a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" is_pre_election: true
I20260620 00:18:21.350165 8200 raft_consensus.cc:2454] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b7f02ab3881440abe79ebb295000a38 in term 0.
I20260620 00:18:21.350392 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a896c1e1be5042e395f369da517bf3f4" candidate_uuid: "1b7f02ab3881440abe79ebb295000a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d87d9c06e364abda3c8cdc8fa885787" is_pre_election: true
I20260620 00:18:21.350414 8271 leader_election.cc:302] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7f02ab3881440abe79ebb295000a38, a61f3b0f50bc4e6db51ebe4acf8d721e; no voters:
I20260620 00:18:21.350489 8060 raft_consensus.cc:2454] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1b7f02ab3881440abe79ebb295000a38 in term 0.
I20260620 00:18:21.350536 8418 raft_consensus.cc:2781] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:21.350579 8418 raft_consensus.cc:482] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:21.350612 8418 raft_consensus.cc:3037] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:21.351306 8418 raft_consensus.cc:504] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.351460 8418 leader_election.cc:288] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [CANDIDATE]: Term 1 election: Requested vote from peers 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723), a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547)
I20260620 00:18:21.351629 8060 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a896c1e1be5042e395f369da517bf3f4" candidate_uuid: "1b7f02ab3881440abe79ebb295000a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d87d9c06e364abda3c8cdc8fa885787"
I20260620 00:18:21.351652 8200 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a896c1e1be5042e395f369da517bf3f4" candidate_uuid: "1b7f02ab3881440abe79ebb295000a38" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e"
I20260620 00:18:21.351783 8200 raft_consensus.cc:3037] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:21.351783 8060 raft_consensus.cc:3037] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:21.352404 8200 raft_consensus.cc:2454] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b7f02ab3881440abe79ebb295000a38 in term 1.
I20260620 00:18:21.352533 8060 raft_consensus.cc:2454] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1b7f02ab3881440abe79ebb295000a38 in term 1.
I20260620 00:18:21.352610 8271 leader_election.cc:302] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1b7f02ab3881440abe79ebb295000a38, a61f3b0f50bc4e6db51ebe4acf8d721e; no voters:
I20260620 00:18:21.352710 8418 raft_consensus.cc:2781] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:21.352867 8418 raft_consensus.cc:686] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 1 LEADER]: Becoming Leader. State: Replica: 1b7f02ab3881440abe79ebb295000a38, State: Running, Role: LEADER
I20260620 00:18:21.352952 8418 consensus_queue.cc:237] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } }
I20260620 00:18:21.353636 7902 catalog_manager.cc:5495] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1b7f02ab3881440abe79ebb295000a38" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1b7f02ab3881440abe79ebb295000a38" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35389 } health_report { overall_health: HEALTHY } } }
I20260620 00:18:21.375252 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.372832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42682 (local address 127.0.98.3:35389)
0620 00:18:21.372940 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:21.372942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:21.372949 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:18:21.372986 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:21.372987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:21.373004 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:21.373093 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:21.373198 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.373710 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.374441 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.374655 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.374746 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.374907 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:21.375020 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:21.375109 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:21.375134 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 00:18:21.375154 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":51,"threads_started":1}
I20260620 00:18:21.379892 8462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.377607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53565 (local address 127.0.98.1:41723)
0620 00:18:21.377684 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:21.377686 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:21.377756 (+ 70us) server_negotiation.cc:407] Connection header received
0620 00:18:21.377808 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:21.377808 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:21.377816 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:21.377890 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:21.378019 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.378463 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.379245 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.379427 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.379482 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.379679 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:21.379715 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:21.379764 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:21.379813 (+ 49us) server_negotiation.cc:299] Negotiation successful
0620 00:18:21.379832 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 00:18:21.379943 8514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.377515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41723 (local address 127.0.98.3:53565)
0620 00:18:21.377677 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:21.377687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:21.377767 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:21.377905 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:21.377906 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:21.377907 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:18:21.377962 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:21.377963 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.378527 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.378528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:21.379166 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:21.379168 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.379496 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.379497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:21.379547 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.379559 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:18:21.379776 (+ 217us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:18:21.379777 (+ 1us) client_negotiation.cc:769] Sending connection context
0620 00:18:21.379812 (+ 35us) client_negotiation.cc:240] Negotiation successful
0620 00:18:21.379849 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260620 00:18:21.381922 8200 raft_consensus.cc:1261] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Refusing update from remote peer 1b7f02ab3881440abe79ebb295000a38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:21.381942 8060 raft_consensus.cc:1261] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Refusing update from remote peer 1b7f02ab3881440abe79ebb295000a38: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:21.382212 8509 consensus_queue.cc:1035] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d87d9c06e364abda3c8cdc8fa885787" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:21.382335 8418 consensus_queue.cc:1035] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a61f3b0f50bc4e6db51ebe4acf8d721e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42547 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:21.389775 8511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.387334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49809 (local address 127.0.98.3:35389)
0620 00:18:21.387376 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:21.387378 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:21.387386 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:18:21.387450 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:21.387451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:21.387462 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:21.387515 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:21.387671 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.388302 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.389077 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.389306 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.389375 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.389548 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:21.389583 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:21.389625 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:21.389685 (+ 60us) server_negotiation.cc:299] Negotiation successful
0620 00:18:21.389706 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:18:21.389801 8478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.387273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35389 (local address 127.0.98.1:49809)
0620 00:18:21.387335 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:21.387348 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:21.387387 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:21.387555 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:21.387557 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:21.387558 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0620 00:18:21.387607 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:21.387608 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.388385 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.388386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:21.389010 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:21.389013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:21.389400 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:21.389401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:21.389470 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:21.389478 (+ 8us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0620 00:18:21.389639 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0620 00:18:21.389642 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:21.389685 (+ 43us) client_negotiation.cc:240] Negotiation successful
0620 00:18:21.389730 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:18:21.396328 7902 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:21.396456 7902 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:21.397061 7902 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2daad448310d4c128e14a0753b386267: Sending DeleteTablet for 3 replicas of tablet a896c1e1be5042e395f369da517bf3f4
I20260620 00:18:21.397356 8180 tablet_service.cc:1514] Processing DeleteTablet for tablet a896c1e1be5042e395f369da517bf3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60174
I20260620 00:18:21.397469 8481 tablet_replica.cc:330] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: stopping tablet replica
I20260620 00:18:21.397477 8040 tablet_service.cc:1514] Processing DeleteTablet for tablet a896c1e1be5042e395f369da517bf3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54986
I20260620 00:18:21.397521 8481 raft_consensus.cc:2229] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:21.397559 8481 raft_consensus.cc:2258] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:21.397626 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 7975
I20260620 00:18:21.397699 8482 tablet_replica.cc:330] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787: stopping tablet replica
I20260620 00:18:21.397753 8482 raft_consensus.cc:2229] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:21.397792 8482 raft_consensus.cc:2258] T a896c1e1be5042e395f369da517bf3f4 P 8d87d9c06e364abda3c8cdc8fa885787 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:21.397938 8481 ts_tablet_manager.cc:1905] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:21.399199 8320 tablet_service.cc:1514] Processing DeleteTablet for tablet a896c1e1be5042e395f369da517bf3f4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:21 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42672
I20260620 00:18:21.399326 8480 tablet_replica.cc:330] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: stopping tablet replica
I20260620 00:18:21.399405 8480 raft_consensus.cc:2229] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:18:21.399495 8480 raft_consensus.cc:2258] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:21.399789 8480 ts_tablet_manager.cc:1905] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:21.401121 8480 ts_tablet_manager.cc:1918] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:18:21.401168 8480 log.cc:1196] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/a896c1e1be5042e395f369da517bf3f4
I20260620 00:18:21.401328 8480 ts_tablet_manager.cc:1939] T a896c1e1be5042e395f369da517bf3f4 P 1b7f02ab3881440abe79ebb295000a38: Deleting consensus metadata
I20260620 00:18:21.401754 7890 catalog_manager.cc:4841] TS 1b7f02ab3881440abe79ebb295000a38 (127.0.98.3:35389): tablet a896c1e1be5042e395f369da517bf3f4 (table test-table [id=2c5e038429a843deb6f957edf4cf19c5]) successfully deleted
I20260620 00:18:21.404517 8481 ts_tablet_manager.cc:1918] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260620 00:18:21.404618 8481 log.cc:1196] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/a896c1e1be5042e395f369da517bf3f4
I20260620 00:18:21.404875 8481 ts_tablet_manager.cc:1939] T a896c1e1be5042e395f369da517bf3f4 P a61f3b0f50bc4e6db51ebe4acf8d721e: Deleting consensus metadata
I20260620 00:18:21.405251 7887 catalog_manager.cc:4841] TS a61f3b0f50bc4e6db51ebe4acf8d721e (127.0.98.2:42547): tablet a896c1e1be5042e395f369da517bf3f4 (table test-table [id=2c5e038429a843deb6f957edf4cf19c5]) successfully deleted
W20260620 00:18:21.406404 7890 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41723 (error 108)
I20260620 00:18:21.406616 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 8115
I20260620 00:18:21.412531 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:21.406898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41723 (local address 127.0.0.1:55016)
0620 00:18:21.412363 (+ 5465us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:21.412465 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41723: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5438,"thread_start_us":37,"threads_started":1}
W20260620 00:18:21.412678 7890 catalog_manager.cc:4584] TS 8d87d9c06e364abda3c8cdc8fa885787 (127.0.98.1:41723): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a896c1e1be5042e395f369da517bf3f4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41723: connect: Connection refused (error 111)
I20260620 00:18:21.413144 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 8255
I20260620 00:18:21.418253 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 7871
2026-06-20T00:18:21Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3149 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:21 dist-test-slave-r81b krb5kdc[8523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:21 dist-test-slave-r81b krb5kdc[8523](info): set up 2 sockets
Jun 20 00:18:21 dist-test-slave-r81b krb5kdc[8523](info): commencing operation
krb5kdc: starting...
W20260620 00:18:22.464968 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:22 dist-test-slave-r81b krb5kdc[8523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:18:22.585142 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45177
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:22.668813 8536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:22.669046 8536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:22.669077 8536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:22.670511 8536 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:22.670560 8536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:22.670575 8536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:22.670588 8536 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:22.670600 8536 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:22.672300 8536 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:22.672582 8536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:22.672837 8536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:22.678027 8544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:22.678054 8542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:22.678023 8541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:22.678375 8536 server_base.cc:1034] running on GCE node
I20260620 00:18:22.678630 8536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:22.678931 8536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:22.680099 8536 hybrid_clock.cc:648] HybridClock initialized: now 1781914702680074 us; error 40 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-06-20T00:18:22Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1263 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:22 dist-test-slave-r81b krb5kdc[8549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:22 dist-test-slave-r81b krb5kdc[8549](info): set up 2 sockets
Jun 20 00:18:22 dist-test-slave-r81b krb5kdc[8549](info): commencing operation
krb5kdc: starting...
W20260620 00:18:23.728868 392 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:23 dist-test-slave-r81b krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:18:23.848271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41543
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:23.930998 8562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:23.931241 8562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:23.931264 8562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:23.932479 8562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:23.932521 8562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:23.932535 8562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:23.932548 8562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:23.932560 8562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:23.934166 8562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:23.934451 8562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:23.934700 8562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:23.939826 8570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:23.939772 8568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:23.939786 8567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:23.940295 8562 server_base.cc:1034] running on GCE node
I20260620 00:18:23.940560 8562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:23.940865 8562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:23.942030 8562 hybrid_clock.cc:648] HybridClock initialized: now 1781914703942019 us; error 48 us; skew 500 ppm
Jun 20 00:18:23 dist-test-slave-r81b krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914703, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:23.944665 8562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:18:23.945125 8562 webserver.cc:460] Webserver started at http://127.0.98.62:34177/ using document root <none> and password file <none>
I20260620 00:18:23.945384 8562 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:23.945432 8562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:23.945526 8562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:23.946650 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "89ef76526b884d2a9aa422d9ecb1013e"
format_stamp: "Formatted at 2026-06-20 00:18:23 on dist-test-slave-r81b"
server_key: "9c77811419f071a3d2f517a8998ab885"
server_key_iv: "ca548981ee5a0946801cc563eaa9f90a"
server_key_version: "encryptionkey@0"
I20260620 00:18:23.947064 8562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "89ef76526b884d2a9aa422d9ecb1013e"
format_stamp: "Formatted at 2026-06-20 00:18:23 on dist-test-slave-r81b"
server_key: "9c77811419f071a3d2f517a8998ab885"
server_key_iv: "ca548981ee5a0946801cc563eaa9f90a"
server_key_version: "encryptionkey@0"
I20260620 00:18:23.948637 8562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:23.949497 8577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:23.949738 8562 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:23.949854 8562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "89ef76526b884d2a9aa422d9ecb1013e"
format_stamp: "Formatted at 2026-06-20 00:18:23 on dist-test-slave-r81b"
server_key: "9c77811419f071a3d2f517a8998ab885"
server_key_iv: "ca548981ee5a0946801cc563eaa9f90a"
server_key_version: "encryptionkey@0"
I20260620 00:18:23.949934 8562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:23.962597 8562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:23.964763 8562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:23.964933 8562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:23.969094 8562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41543
I20260620 00:18:23.969147 8629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41543 every 8 connection(s)
I20260620 00:18:23.969687 8562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:23.970420 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:23.972982 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: Bootstrap starting.
I20260620 00:18:23.973338 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 8562
I20260620 00:18:23.973459 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:23.973613 8630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:23.973673 392 external_mini_cluster.cc:1428] Setting key b65dab3e33da5b89f8df3d82b3a092af
I20260620 00:18:23.973917 8630 log.cc:824] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:23.974728 8630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: No bootstrap required, opened a new log
I20260620 00:18:23.976871 8630 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } }
I20260620 00:18:23.977042 8630 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:23.977101 8630 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ef76526b884d2a9aa422d9ecb1013e, State: Initialized, Role: FOLLOWER
I20260620 00:18:23.977260 8630 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } }
I20260620 00:18:23.977344 8630 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:23.977386 8630 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:23.977434 8630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:23.977995 8630 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } }
I20260620 00:18:23.978124 8630 leader_election.cc:302] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ef76526b884d2a9aa422d9ecb1013e; no voters:
I20260620 00:18:23.978327 8630 leader_election.cc:288] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:23.978355 8635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:23.978461 8635 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 1 LEADER]: Becoming Leader. State: Replica: 89ef76526b884d2a9aa422d9ecb1013e, State: Running, Role: LEADER
I20260620 00:18:23.978542 8635 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } }
I20260620 00:18:23.978596 8630 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:23.978883 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89ef76526b884d2a9aa422d9ecb1013e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } } }
I20260620 00:18:23.978952 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:23.979136 8635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ef76526b884d2a9aa422d9ecb1013e. Latest consensus state: current_term: 1 leader_uuid: "89ef76526b884d2a9aa422d9ecb1013e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } } }
I20260620 00:18:23.979223 8640 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:23.979286 8635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:23.979825 8640 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:23.981709 8640 catalog_manager.cc:1347] Generated new cluster ID: c2e6a6e4336844b98eca3a56028660a7
I20260620 00:18:23.981763 8640 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:18:23 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914703, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:23.986178 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:23.974210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53056 (local address 127.0.98.62:41543)
0620 00:18:23.974384 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:23.974395 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:23.974433 (+ 38us) server_negotiation.cc:407] Connection header received
0620 00:18:23.974944 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:23.974951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:23.975010 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:23.975171 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:23.975820 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:23.976817 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:23.977831 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:23.978346 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:23.978471 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:23.982558 (+ 4087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:23.982562 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:23.982562 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:23.982575 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:23.984420 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:23.984813 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:23.984816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:23.984817 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:23.984911 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:23.985142 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:23.985143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:23.985143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:23.985392 (+ 249us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:23.985576 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:23.985666 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 00:18:23.985809 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":77,"threads_started":1}
I20260620 00:18:23.993994 8640 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:23.994644 8640 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:24.003667 8640 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: Generated new TSK 0
I20260620 00:18:24.003937 8640 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:18:24.035090 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41543
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:24.132068 8656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:24.132314 8656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:24.132339 8656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:24.133487 8656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:24.133525 8656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:24.133550 8656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:24.135069 8656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:24.135327 8656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:24.135646 8656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:18:24.141064 8656 server_base.cc:1034] running on GCE node
W20260620 00:18:24.141037 8662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:24.141036 8661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:24.141037 8664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:24.141479 8656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:24.141763 8656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:24.142959 8656 hybrid_clock.cc:648] HybridClock initialized: now 1781914704142939 us; error 40 us; skew 500 ppm
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:24.146028 8656 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:18:24.146606 8656 webserver.cc:460] Webserver started at http://127.0.98.1:40703/ using document root <none> and password file <none>
I20260620 00:18:24.146894 8656 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:24.146968 8656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:24.147132 8656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:24.148360 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "d9a6081e3b014def97f12f9286b6ab1f"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "85341e752c24a97396cfc1e25f9d07a1"
server_key_iv: "31cff61f001f09a7d1a6d21002f24371"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.148746 8656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d9a6081e3b014def97f12f9286b6ab1f"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "85341e752c24a97396cfc1e25f9d07a1"
server_key_iv: "31cff61f001f09a7d1a6d21002f24371"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.150170 8656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:18:24.151010 8671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.151240 8656 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:24.151372 8656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "d9a6081e3b014def97f12f9286b6ab1f"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "85341e752c24a97396cfc1e25f9d07a1"
server_key_iv: "31cff61f001f09a7d1a6d21002f24371"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.151461 8656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:24.180363 8656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:24.183025 8656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:24.183198 8656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:24.183454 8656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:24.183781 8656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:24.183840 8656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.183883 8656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:24.183911 8656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.189935 8656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43745
I20260620 00:18:24.190037 8784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43745 every 8 connection(s)
I20260620 00:18:24.190387 8656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:18:24.191148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 8656
I20260620 00:18:24.191262 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:24.191509 392 external_mini_cluster.cc:1428] Setting key af1e345f060e8359bce5ebc875b72d8b
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:18:24.201653 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:24.191157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37339 (local address 127.0.98.62:41543)
0620 00:18:24.191412 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:24.191415 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:24.191476 (+ 61us) server_negotiation.cc:407] Connection header received
0620 00:18:24.192696 (+ 1220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:24.192697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:24.192708 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:24.192762 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:24.193673 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.194222 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.195224 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.195439 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.195530 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.198276 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:24.198278 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:24.198278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:24.198287 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:24.199959 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:24.200516 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:24.200517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:24.200518 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:24.200575 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:24.200934 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:24.200935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:24.200936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:24.201143 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:24.201209 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:24.201490 (+ 281us) server_negotiation.cc:299] Negotiation successful
0620 00:18:24.201579 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":225}
I20260620 00:18:24.201797 8787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:24.191118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:37339)
0620 00:18:24.191337 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:24.191392 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:24.192584 (+ 1192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:24.192846 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:24.192850 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:24.193254 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:24.193545 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.193552 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.194317 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.194318 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.195126 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.195128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.195542 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.195543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.195680 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.196327 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:24.196338 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:24.198164 (+ 1826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:24.200063 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:24.200073 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:24.200076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:24.200405 (+ 329us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:24.200705 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:24.200706 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:24.200707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:24.200824 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:24.201220 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:24.201225 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:24.201361 (+ 136us) client_negotiation.cc:769] Sending connection context
0620 00:18:24.201447 (+ 86us) client_negotiation.cc:240] Negotiation successful
0620 00:18:24.201581 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":126,"threads_started":1}
I20260620 00:18:24.202430 8785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41543
I20260620 00:18:24.202543 8785 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:18:24.202807 8785 heartbeater.cc:511] Master 127.0.98.62:41543 requested a full tablet report, sending...
I20260620 00:18:24.203392 8594 ts_manager.cc:194] Registered new tserver with Master: d9a6081e3b014def97f12f9286b6ab1f (127.0.98.1:43745)
I20260620 00:18:24.204020 8594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37339
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:18:24.219626 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41543
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:24.304116 8790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:24.304407 8790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:24.304433 8790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:24.305650 8790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:24.305696 8790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:24.305722 8790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:24.307266 8790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:24.307526 8790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:24.307776 8790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:24.313076 8798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:24.313104 8796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:24.313097 8795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:24.313397 8790 server_base.cc:1034] running on GCE node
I20260620 00:18:24.313621 8790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:24.313905 8790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:24.315079 8790 hybrid_clock.cc:648] HybridClock initialized: now 1781914704315054 us; error 43 us; skew 500 ppm
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:24.317982 8790 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:18:24.318480 8790 webserver.cc:460] Webserver started at http://127.0.98.2:36743/ using document root <none> and password file <none>
I20260620 00:18:24.318694 8790 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:24.318742 8790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:24.318847 8790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:24.319970 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "255baa1f57ee4cd4ad7d1d47b0eaa208"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "31a88e869cd0ac520f6de2fc2e6e06ea"
server_key_iv: "d034d7a71dd1dee5b80906253f527af1"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.320410 8790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "255baa1f57ee4cd4ad7d1d47b0eaa208"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "31a88e869cd0ac520f6de2fc2e6e06ea"
server_key_iv: "d034d7a71dd1dee5b80906253f527af1"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.322634 8790 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 00:18:24.323619 8805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.323843 8790 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:24.323954 8790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "255baa1f57ee4cd4ad7d1d47b0eaa208"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "31a88e869cd0ac520f6de2fc2e6e06ea"
server_key_iv: "d034d7a71dd1dee5b80906253f527af1"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.324062 8790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:24.361017 8790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:24.363250 8790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:24.363384 8790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:24.363591 8790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:24.363880 8790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:24.363915 8790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.363940 8790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:24.363955 8790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.370565 8790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40527
I20260620 00:18:24.370605 8918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40527 every 8 connection(s)
I20260620 00:18:24.370976 8790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:24.375129 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 8790
I20260620 00:18:24.375222 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:24.375415 392 external_mini_cluster.cc:1428] Setting key 1b82a4acb6fa86782547c8d604442cc0
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:24.382726 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:24.371823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33641 (local address 127.0.98.62:41543)
0620 00:18:24.371887 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:24.371890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:24.372182 (+ 292us) server_negotiation.cc:407] Connection header received
0620 00:18:24.373401 (+ 1219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:24.373403 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:24.373418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:24.373508 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:24.374191 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.374842 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.376390 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.376700 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.376782 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.379307 (+ 2525us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:24.379309 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:24.379310 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:24.379318 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:24.381148 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:24.381595 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:24.381597 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:24.381597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:24.381666 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:24.381919 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:24.381920 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:24.381920 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:24.382105 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:24.382190 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:24.382510 (+ 320us) server_negotiation.cc:299] Negotiation successful
0620 00:18:24.382614 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:18:24.382848 8921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:24.371819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:33641)
0620 00:18:24.372012 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:24.372055 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:24.373219 (+ 1164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:24.373576 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:24.373580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:24.373873 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:24.374088 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.374092 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.374931 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.374931 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.376299 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.376305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.376811 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.376812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.376925 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.377535 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:24.377544 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:24.379210 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:24.381250 (+ 2040us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:24.381257 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:24.381259 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:24.381513 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:24.381740 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:24.381740 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:24.381741 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:24.381847 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:24.382202 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:24.382209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:24.382356 (+ 147us) client_negotiation.cc:769] Sending connection context
0620 00:18:24.382460 (+ 104us) client_negotiation.cc:240] Negotiation successful
0620 00:18:24.382606 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":101,"threads_started":1}
I20260620 00:18:24.383441 8919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41543
I20260620 00:18:24.383574 8919 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:24.383777 8919 heartbeater.cc:511] Master 127.0.98.62:41543 requested a full tablet report, sending...
I20260620 00:18:24.384302 8594 ts_manager.cc:194] Registered new tserver with Master: 255baa1f57ee4cd4ad7d1d47b0eaa208 (127.0.98.2:40527)
I20260620 00:18:24.384847 8594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33641
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:18:24.404562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41543
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:24.488127 8924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:24.488425 8924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:24.488451 8924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:24.489708 8924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:24.489755 8924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:24.489782 8924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:24.491288 8924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:24.491488 8924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:24.491667 8924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:18:24.496953 8924 server_base.cc:1034] running on GCE node
W20260620 00:18:24.496933 8929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:24.496915 8930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:24.496973 8932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:24.497418 8924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:24.497735 8924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:24.498919 8924 hybrid_clock.cc:648] HybridClock initialized: now 1781914704498922 us; error 46 us; skew 500 ppm
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:24.501686 8924 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:18:24.502116 8924 webserver.cc:460] Webserver started at http://127.0.98.3:39215/ using document root <none> and password file <none>
I20260620 00:18:24.502308 8924 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:24.502350 8924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:24.502444 8924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:24.503567 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "c49303fd0cc94e30b5658a0059863ebc"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "f5b4e758a71d84ccd6dcffcb21cbd354"
server_key_iv: "435bc8597c57b70888f4b30bf0804b0f"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.503942 8924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c49303fd0cc94e30b5658a0059863ebc"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "f5b4e758a71d84ccd6dcffcb21cbd354"
server_key_iv: "435bc8597c57b70888f4b30bf0804b0f"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.505378 8924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:24.506181 8939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.506394 8924 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:24.506487 8924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "c49303fd0cc94e30b5658a0059863ebc"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "f5b4e758a71d84ccd6dcffcb21cbd354"
server_key_iv: "435bc8597c57b70888f4b30bf0804b0f"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.506547 8924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:24.515133 8924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:24.517189 8924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:24.517313 8924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:24.517501 8924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:24.517819 8924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:24.517854 8924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.517901 8924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:24.517928 8924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.523712 8924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43721
I20260620 00:18:24.523830 9052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43721 every 8 connection(s)
I20260620 00:18:24.524127 8924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:24.529577 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 8924
I20260620 00:18:24.529663 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:24.529875 392 external_mini_cluster.cc:1428] Setting key df9ecd728d37aee6fcf6d5e10be1f97e
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:24.533762 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:24.525060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40399 (local address 127.0.98.62:41543)
0620 00:18:24.525127 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:24.525130 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:24.525366 (+ 236us) server_negotiation.cc:407] Connection header received
0620 00:18:24.526287 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:24.526288 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:24.526299 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:24.526347 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:24.526955 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.527454 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.528445 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.528655 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.528710 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.531279 (+ 2569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:24.531280 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:24.531281 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:24.531288 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:24.532572 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:24.532943 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:24.532943 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:24.532944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:24.532993 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:24.533225 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:24.533226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:24.533226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:24.533332 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:24.533385 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:24.533627 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 00:18:24.533689 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:18:24.533946 9055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:24.525045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:40399)
0620 00:18:24.525244 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:24.525280 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:24.526193 (+ 913us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:24.526411 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:24.526414 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:24.526683 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:24.526874 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.526878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.527532 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.527533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.528358 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.528359 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.528739 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.528739 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.528884 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.529413 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:24.529422 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:24.531186 (+ 1764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:24.532646 (+ 1460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:24.532651 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:24.532653 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:24.532875 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:24.533049 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:24.533049 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:24.533049 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:24.533149 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:24.533388 (+ 239us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:24.533392 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:24.533509 (+ 117us) client_negotiation.cc:769] Sending connection context
0620 00:18:24.533592 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 00:18:24.533712 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":113,"threads_started":1}
I20260620 00:18:24.534456 9053 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41543
I20260620 00:18:24.534538 9053 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:24.534740 9053 heartbeater.cc:511] Master 127.0.98.62:41543 requested a full tablet report, sending...
I20260620 00:18:24.535091 8594 ts_manager.cc:194] Registered new tserver with Master: c49303fd0cc94e30b5658a0059863ebc (127.0.98.3:43721)
I20260620 00:18:24.535511 8594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40399
I20260620 00:18:24.541841 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260620 00:18:24.616855 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 8656
I20260620 00:18:24.621488 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 8790
I20260620 00:18:24.625841 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 8924
I20260620 00:18:24.630008 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 8562
I20260620 00:18:24.634991 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41543
--webserver_interface=127.0.98.62
--webserver_port=34177
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:24.718094 9062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:24.718372 9062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:24.718410 9062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:24.719631 9062 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:24.719703 9062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:24.719727 9062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:24.719749 9062 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:24.719779 9062 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:24.721416 9062 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=34177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9062
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:24.721715 9062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:24.721958 9062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:24.726953 9067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:24.726948 9070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:24.727026 9062 server_base.cc:1034] running on GCE node
W20260620 00:18:24.726972 9068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:24.727499 9062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:24.727779 9062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:24.728950 9062 hybrid_clock.cc:648] HybridClock initialized: now 1781914704728944 us; error 38 us; skew 500 ppm
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914704, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:24.736614 9062 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:18:24.737159 9062 webserver.cc:460] Webserver started at http://127.0.98.62:34177/ using document root <none> and password file <none>
I20260620 00:18:24.737418 9062 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:24.737511 9062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:24.739012 9062 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:24.739862 9077 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.740094 9062 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:24.740233 9062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "89ef76526b884d2a9aa422d9ecb1013e"
format_stamp: "Formatted at 2026-06-20 00:18:23 on dist-test-slave-r81b"
server_key: "9c77811419f071a3d2f517a8998ab885"
server_key_iv: "ca548981ee5a0946801cc563eaa9f90a"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.740610 9062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:24.751499 9062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:24.753666 9062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:24.753856 9062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:24.758152 9062 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41543
I20260620 00:18:24.758185 9129 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41543 every 8 connection(s)
I20260620 00:18:24.758746 9062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:24.760001 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 9062
I20260620 00:18:24.760221 9130 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:18:24.760378 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43745
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41543
--webserver_port=40703
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:18:24.761313 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: Bootstrap starting.
I20260620 00:18:24.769775 9130 log.cc:824] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:24.771646 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:18:24.771906 9130 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: Bootstrap complete.
I20260620 00:18:24.774494 9130 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } }
I20260620 00:18:24.774736 9130 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89ef76526b884d2a9aa422d9ecb1013e, State: Initialized, Role: FOLLOWER
I20260620 00:18:24.774897 9130 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } }
I20260620 00:18:24.774973 9130 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:24.775002 9130 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:24.775063 9130 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:18:24.776257 9130 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } }
I20260620 00:18:24.776412 9130 leader_election.cc:302] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89ef76526b884d2a9aa422d9ecb1013e; no voters:
I20260620 00:18:24.776635 9130 leader_election.cc:288] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:18:24.776698 9134 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:18:24.776921 9130 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:24.776907 9134 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [term 2 LEADER]: Becoming Leader. State: Replica: 89ef76526b884d2a9aa422d9ecb1013e, State: Running, Role: LEADER
I20260620 00:18:24.777140 9134 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } }
I20260620 00:18:24.777518 9135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "89ef76526b884d2a9aa422d9ecb1013e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } } }
I20260620 00:18:24.777638 9135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:24.777820 9136 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89ef76526b884d2a9aa422d9ecb1013e. Latest consensus state: current_term: 2 leader_uuid: "89ef76526b884d2a9aa422d9ecb1013e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89ef76526b884d2a9aa422d9ecb1013e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41543 } } }
I20260620 00:18:24.777918 9136 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:24.778044 9144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:24.778581 9144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:24.778760 9144 catalog_manager.cc:1259] Loaded cluster ID: c2e6a6e4336844b98eca3a56028660a7
I20260620 00:18:24.778928 9144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:24.780189 9144 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:24.780376 9144 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 89ef76526b884d2a9aa422d9ecb1013e: Loaded TSK: 0
I20260620 00:18:24.780582 9144 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:18:24.846557 9132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:24.846834 9132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:24.846879 9132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:24.848289 9132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:24.848337 9132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:24.848367 9132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:24.849920 9132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43745
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=40703
--tserver_master_addrs=127.0.98.62:41543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:24.850186 9132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:24.850437 9132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:24.856534 9157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:24.856550 9156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:24.856796 9132 server_base.cc:1034] running on GCE node
W20260620 00:18:24.856840 9159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:24.857136 9132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:24.857434 9132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:24.858624 9132 hybrid_clock.cc:648] HybridClock initialized: now 1781914704858605 us; error 31 us; skew 500 ppm
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914704, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:24.861863 9132 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260620 00:18:24.862401 9132 webserver.cc:460] Webserver started at http://127.0.98.1:40703/ using document root <none> and password file <none>
I20260620 00:18:24.862653 9132 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:24.862767 9132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:24.864274 9132 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:24.867892 9166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.868094 9132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 00:18:24.868254 9132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "d9a6081e3b014def97f12f9286b6ab1f"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "85341e752c24a97396cfc1e25f9d07a1"
server_key_iv: "31cff61f001f09a7d1a6d21002f24371"
server_key_version: "encryptionkey@0"
I20260620 00:18:24.868646 9132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:24.893676 9132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:24.896124 9132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:24.896360 9132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:24.896620 9132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:24.896991 9132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:24.897043 9132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.897076 9132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:24.897104 9132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:24.905383 9132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43745
I20260620 00:18:24.905418 9279 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43745 every 8 connection(s)
I20260620 00:18:24.905906 9132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:24.915397 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:24.906805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:59977)
0620 00:18:24.906996 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:24.907043 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:24.907512 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:24.907962 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:24.907965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:24.908623 (+ 658us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:24.908819 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.908824 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.910365 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.910366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.911185 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.911187 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.911966 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.911967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.912086 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.912762 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:24.912770 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:24.915206 (+ 2436us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":94,"threads_started":1}
W20260620 00:18:24.915503 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:24.906857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59977 (local address 127.0.98.62:41543)
0620 00:18:24.907095 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:24.907104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:24.907145 (+ 41us) server_negotiation.cc:407] Connection header received
0620 00:18:24.907715 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:24.907723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:24.907779 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:24.907920 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:24.908940 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.910243 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.911279 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.911879 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.911980 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.915341 (+ 3361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59977: BlockingRecv error: recv got EOF from 127.0.98.1:59977 (error 108)
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":105,"threads_started":1}
W20260620 00:18:24.915686 9280 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41543 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41543: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260620 00:18:24.916107 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 9132
I20260620 00:18:24.916412 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40527
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41543
--webserver_port=36743
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:24.921797 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:24.915944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:58481)
0620 00:18:24.916239 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:24.916252 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:24.916303 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:24.916614 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:24.916615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:24.916757 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:24.916807 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.916808 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.917563 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.917565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.918449 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.918451 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.918956 (+ 505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.918958 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.919026 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.919577 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:24.919581 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:24.921739 (+ 2158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":273}
W20260620 00:18:24.921952 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:24.916391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58481 (local address 127.0.98.62:41543)
0620 00:18:24.916443 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:24.916444 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:24.916454 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:24.916480 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:24.916481 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:24.916494 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:24.916554 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:24.916888 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.917449 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.918567 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.918843 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.918918 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.921888 (+ 2970us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:58481: BlockingRecv error: recv got EOF from 127.0.98.1:58481 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:24 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:24.927241 9282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:24.922107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:56261)
0620 00:18:24.922161 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:24.922173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:24.922269 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:24.922436 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:24.922436 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:24.922567 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:24.922606 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.922608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.923672 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.923673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.924277 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:24.924280 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.924774 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.924775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:24.924812 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.925315 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:24.925319 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:24.927135 (+ 1816us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:18:24.927342 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:24.922229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56261 (local address 127.0.98.62:41543)
0620 00:18:24.922299 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:24.922301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:24.922313 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:18:24.922341 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:24.922342 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:24.922355 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:24.922427 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:24.922703 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.923576 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.924357 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:24.924691 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:24.924763 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:24.927232 (+ 2469us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:56261: BlockingRecv error: recv got EOF from 127.0.98.1:56261 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260620 00:18:24.927399 9280 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:18:25.000952 9284 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:25.001184 9284 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:25.001209 9284 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:25.002345 9284 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:25.002389 9284 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:25.002416 9284 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:25.003808 9284 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40527
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=36743
--tserver_master_addrs=127.0.98.62:41543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9284
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:25.004004 9284 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:25.004272 9284 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:25.009379 9289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:25.009413 9292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:25.009467 9284 server_base.cc:1034] running on GCE node
W20260620 00:18:25.009409 9290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:25.009843 9284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:25.010126 9284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:25.011309 9284 hybrid_clock.cc:648] HybridClock initialized: now 1781914705011294 us; error 28 us; skew 500 ppm
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:25.013967 9284 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260620 00:18:25.014472 9284 webserver.cc:460] Webserver started at http://127.0.98.2:36743/ using document root <none> and password file <none>
I20260620 00:18:25.014703 9284 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:25.014783 9284 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:25.016242 9284 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:25.016887 9299 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:25.017055 9284 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:25.017165 9284 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "255baa1f57ee4cd4ad7d1d47b0eaa208"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "31a88e869cd0ac520f6de2fc2e6e06ea"
server_key_iv: "d034d7a71dd1dee5b80906253f527af1"
server_key_version: "encryptionkey@0"
I20260620 00:18:25.017525 9284 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:25.036386 9284 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:25.038789 9284 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:25.038954 9284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:25.039209 9284 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:25.039526 9284 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:25.039578 9284 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:25.039623 9284 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:25.039651 9284 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:25.045933 9284 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40527
I20260620 00:18:25.045962 9412 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40527 every 8 connection(s)
I20260620 00:18:25.046343 9284 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:25.052431 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 9284
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:25.052754 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43721
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41543
--webserver_port=39215
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260620 00:18:25.054567 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:25.047076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42451 (local address 127.0.98.62:41543)
0620 00:18:25.047162 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:25.047164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:25.047416 (+ 252us) server_negotiation.cc:407] Connection header received
0620 00:18:25.047937 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:25.047938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:25.047953 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:25.048042 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:25.049065 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.049613 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.050671 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.050862 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.050927 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.054510 (+ 3583us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:42451: BlockingRecv error: recv got EOF from 127.0.98.2:42451 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 00:18:25.054621 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:25.047094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:42451)
0620 00:18:25.047257 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:25.047301 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:25.047816 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:25.048108 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:25.048112 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:25.048727 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:25.048942 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.048947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.049717 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.049718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.050570 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.050572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.050941 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.050942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.051045 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.051654 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:25.051663 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:25.054413 (+ 2750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
W20260620 00:18:25.054864 9413 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41543 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41543: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:25.059257 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:25.055091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:55519)
0620 00:18:25.055142 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:25.055154 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:25.055195 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:25.055337 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:25.055338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:25.055472 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:25.055523 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.055524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.056194 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.056195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.056709 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.056710 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.057080 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.057081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.057145 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.057594 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:25.057600 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:25.059204 (+ 1604us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:18:25.059422 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:25.055161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55519 (local address 127.0.98.62:41543)
0620 00:18:25.055207 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:25.055209 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:25.055218 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:25.055254 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:25.055255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:25.055264 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:25.055333 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:25.055602 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.056071 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.056798 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.056996 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.057079 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.059334 (+ 2255us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55519: BlockingRecv error: recv got EOF from 127.0.98.2:55519 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:25.064127 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:25.059571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:39089)
0620 00:18:25.059667 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:25.059676 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:25.059704 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:25.059874 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:25.059875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:25.059997 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:25.060028 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.060029 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.060692 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.060693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.061183 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.061184 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.061538 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.061539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.061589 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.062136 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:25.062140 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:25.064049 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":83}
W20260620 00:18:25.064328 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:25.059697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39089 (local address 127.0.98.62:41543)
0620 00:18:25.059760 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:25.059762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:25.059769 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:18:25.059789 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:25.059790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:25.059799 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:25.059869 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:25.060106 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.060600 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.061265 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.061463 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.061542 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.064261 (+ 2719us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:39089: BlockingRecv error: recv got EOF from 127.0.98.2:39089 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 00:18:25.064425 9413 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:18:25.138593 9416 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:25.138834 9416 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:25.138871 9416 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:25.140239 9416 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:25.140300 9416 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:25.140331 9416 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:25.141997 9416 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43721
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39215
--tserver_master_addrs=127.0.98.62:41543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9416
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:25.142299 9416 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:25.142577 9416 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:25.148058 9424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:25.148118 9422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:25.148204 9416 server_base.cc:1034] running on GCE node
W20260620 00:18:25.148124 9421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:25.148610 9416 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:25.148854 9416 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:25.150041 9416 hybrid_clock.cc:648] HybridClock initialized: now 1781914705150017 us; error 39 us; skew 500 ppm
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914705, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:25.152870 9416 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260620 00:18:25.153359 9416 webserver.cc:460] Webserver started at http://127.0.98.3:39215/ using document root <none> and password file <none>
I20260620 00:18:25.153625 9416 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:25.153704 9416 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:25.155089 9416 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:25.155723 9431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:25.155922 9416 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:25.156028 9416 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "c49303fd0cc94e30b5658a0059863ebc"
format_stamp: "Formatted at 2026-06-20 00:18:24 on dist-test-slave-r81b"
server_key: "f5b4e758a71d84ccd6dcffcb21cbd354"
server_key_iv: "435bc8597c57b70888f4b30bf0804b0f"
server_key_version: "encryptionkey@0"
I20260620 00:18:25.156473 9416 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:25.175873 9416 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:25.178246 9416 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:25.178395 9416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:25.178649 9416 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:25.178973 9416 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:25.179032 9416 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:25.179076 9416 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:25.179103 9416 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:25.184883 9416 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43721
I20260620 00:18:25.184938 9544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43721 every 8 connection(s)
I20260620 00:18:25.185329 9416 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:25.188349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 9416
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:25.193578 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:25.186084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:42507)
0620 00:18:25.186271 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:25.186308 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:25.186778 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:25.186992 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:25.186995 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:25.187589 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:25.187790 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.187796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.188606 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.188607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.189614 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.189617 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.190046 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.190047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.190213 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.190811 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:25.190821 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:25.193407 (+ 2586us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
W20260620 00:18:25.193660 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:25.186116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42507 (local address 127.0.98.62:41543)
0620 00:18:25.186199 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:25.186202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:25.186399 (+ 197us) server_negotiation.cc:407] Connection header received
0620 00:18:25.186871 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:25.186872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:25.186883 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:25.186936 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:25.187882 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.188499 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.189728 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.189957 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.190008 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.193602 (+ 3594us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:42507: BlockingRecv error: recv got EOF from 127.0.98.3:42507 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 00:18:25.193879 9545 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41543 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:41543: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260620 00:18:25.197070 9549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:25.188917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34888 (local address 127.0.98.62:41543)
0620 00:18:25.189044 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:25.189046 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:25.189054 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:18:25.189076 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:25.189077 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:25.189094 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:25.189163 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:25.189817 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.190516 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.191532 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.191764 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.191873 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.193360 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:25.193364 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:25.193364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:25.193497 (+ 133us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:25.195434 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:25.195847 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:25.195851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:25.195851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:25.195955 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:25.196205 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:25.196206 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:25.196207 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:25.196549 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:25.196715 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:25.196826 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 00:18:25.196942 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:25.198359 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:25.194123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:53687)
0620 00:18:25.194234 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:25.194243 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:25.194283 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:25.194417 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:25.194418 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:25.194528 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:25.194572 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.194573 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.195212 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.195213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.195706 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.195707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.196069 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.196070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.196182 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.196566 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:25.196571 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:25.198310 (+ 1739us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98}
W20260620 00:18:25.198495 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:25.194188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53687 (local address 127.0.98.62:41543)
0620 00:18:25.194235 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:25.194237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:25.194274 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:18:25.194322 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:25.194323 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:25.194333 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:25.194395 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:25.194659 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.195117 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.195786 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.195985 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.196068 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.198418 (+ 2350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53687: BlockingRecv error: recv got EOF from 127.0.98.3:53687 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:25.202737 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:25.198613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:47131)
0620 00:18:25.198644 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:25.198654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:25.198689 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:25.198834 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:25.198835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:25.198937 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:25.198966 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.198967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.199575 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.199576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.200098 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.200100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.200552 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.200553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.200596 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.201086 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:25.201091 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:25.202654 (+ 1563us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 00:18:25.202771 9283 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:25.198686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47131 (local address 127.0.98.62:41543)
0620 00:18:25.198734 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:25.198736 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:25.198743 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:18:25.198762 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:25.198763 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:25.198771 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:25.198829 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:25.199020 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.199460 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.200278 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.200452 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.200506 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.202716 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47131: BlockingRecv error: recv got EOF from 127.0.98.3:47131 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 00:18:25.202930 9545 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:25 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:25.933141 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:25.927911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:42061)
0620 00:18:25.928054 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:25.928066 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:25.928126 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:25.928345 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:25.928345 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:25.928509 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:25.928560 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.928561 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.929240 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.929241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.929783 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:25.929784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.930170 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.930172 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:25.930226 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.930849 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:25.930854 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:25.933027 (+ 2173us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
W20260620 00:18:25.933182 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:25.928048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42061 (local address 127.0.98.62:41543)
0620 00:18:25.928196 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:25.928203 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:25.928212 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:25.928237 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:25.928237 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:25.928247 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:25.928328 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:25.928640 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.929158 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.929865 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:25.930058 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:25.930131 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:25.933111 (+ 2980us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:42061: BlockingRecv error: recv got EOF from 127.0.98.1:42061 (error 108)
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
Jun 20 00:18:26 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:26 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:26.069845 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:26.064889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:37153)
0620 00:18:26.065034 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:26.065047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:26.065132 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:26.065265 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:26.065266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:26.065430 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:26.065478 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:26.065479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.066191 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.066192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:26.066823 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:26.066826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.067211 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.067212 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:26.067253 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:26.067793 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:26.067798 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:26.069765 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":68,"threads_started":1}
W20260620 00:18:26.069926 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:26.064990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37153 (local address 127.0.98.62:41543)
0620 00:18:26.065063 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:26.065064 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:26.065119 (+ 55us) server_negotiation.cc:407] Connection header received
0620 00:18:26.065185 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:26.065186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:26.065198 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:26.065252 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:26.065580 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.066093 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.066915 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.067136 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.067201 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:26.069877 (+ 2676us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37153: BlockingRecv error: recv got EOF from 127.0.98.2:37153 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Jun 20 00:18:26 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:26 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:26.208539 9553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:26.203403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:48113)
0620 00:18:26.203554 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:26.203566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:26.203635 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:26.203805 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:26.203806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:26.203987 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:26.204038 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:26.204040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.204797 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.204798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:26.205350 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:26.205351 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.205736 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.205737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:26.205776 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:26.206290 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:26.206294 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:26.208420 (+ 2126us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
W20260620 00:18:26.208705 9551 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:26.203494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48113 (local address 127.0.98.62:41543)
0620 00:18:26.203592 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:26.203593 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:26.203620 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:18:26.203683 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:26.203684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:26.203698 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:26.203780 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:26.204205 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.204718 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.205444 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.205656 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.205727 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:26.208614 (+ 2887us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:48113: BlockingRecv error: recv got EOF from 127.0.98.3:48113 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 00:18:26 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:26 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:26.938802 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:26.933748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:60857)
0620 00:18:26.933885 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:26.933896 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:26.933954 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:26.934164 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:26.934165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:26.934323 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:26.934373 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:26.934374 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.935074 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.935076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:26.935810 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:26.935814 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.936329 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.936330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:26.936390 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:26.936924 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:26.936929 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:26.938695 (+ 1766us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
W20260620 00:18:26.938869 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:26.933895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60857 (local address 127.0.98.62:41543)
0620 00:18:26.934023 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:26.934025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:26.934034 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:26.934062 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:26.934063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:26.934075 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:26.934148 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:26.934440 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.934989 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.935893 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:26.936241 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:26.936316 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:26.938779 (+ 2463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60857: BlockingRecv error: recv got EOF from 127.0.98.1:60857 (error 108)
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":77,"threads_started":1}
Jun 20 00:18:27 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:27 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:27.075883 9556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:27.070518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:33909)
0620 00:18:27.070680 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:27.070695 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:27.070773 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:27.070916 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:27.070917 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:27.071179 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:27.071241 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:27.071243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.072032 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.072033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:27.072800 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:27.072803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.073228 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.073229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:27.073277 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:27.073833 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:27.073838 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:27.075773 (+ 1935us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":81,"threads_started":1}
W20260620 00:18:27.075944 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:27.070539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33909 (local address 127.0.98.62:41543)
0620 00:18:27.070629 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:27.070632 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:27.070772 (+ 140us) server_negotiation.cc:407] Connection header received
0620 00:18:27.070829 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:27.070829 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:27.070845 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:27.070907 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:27.071351 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.071864 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.072902 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.073112 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.073205 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:27.075869 (+ 2664us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33909: BlockingRecv error: recv got EOF from 127.0.98.2:33909 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Jun 20 00:18:27 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:27 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:27.214566 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:27.209217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:59591)
0620 00:18:27.209473 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:27.209485 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:27.209558 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:27.209739 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:27.209740 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:27.209926 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:27.209977 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:27.209978 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.210816 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.210817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:27.211491 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:27.211494 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.211879 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.211880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:27.211929 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:27.212412 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:27.212417 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:27.214499 (+ 2082us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":140,"threads_started":1}
W20260620 00:18:27.214682 9555 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:27.209387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59591 (local address 127.0.98.62:41543)
0620 00:18:27.209461 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:27.209463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:27.209573 (+ 110us) server_negotiation.cc:407] Connection header received
0620 00:18:27.209617 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:27.209618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:27.209630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:27.209691 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:27.210099 (+ 408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.210710 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.211588 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.211790 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.211850 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:27.214606 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59591: BlockingRecv error: recv got EOF from 127.0.98.3:59591 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 00:18:27 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:27 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:27.944657 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:27.939437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:59857)
0620 00:18:27.939591 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:27.939603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:27.939668 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:27.939865 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:27.939866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:27.940067 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:27.940130 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:27.940132 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.940904 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.940906 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:27.941477 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:27.941480 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.941863 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.941864 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:27.941926 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:27.942415 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:27.942422 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:27.944598 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
W20260620 00:18:27.944828 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:27.939536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59857 (local address 127.0.98.62:41543)
0620 00:18:27.939664 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:27.939666 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:27.939677 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:27.939716 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:27.939717 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:27.939727 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:27.939812 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:27.940254 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.940801 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.941572 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:27.941792 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:27.941852 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:27.944713 (+ 2861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59857: BlockingRecv error: recv got EOF from 127.0.98.1:59857 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
Jun 20 00:18:28 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:28 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:28.081647 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:28.076545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:57035)
0620 00:18:28.076785 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:28.076799 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:28.076909 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:28.077123 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:28.077124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:28.077300 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:28.077353 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:28.077354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.078150 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.078151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:28.078758 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:28.078760 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.079123 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.079124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:28.079168 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:28.079618 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:28.079623 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:28.081573 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":116,"threads_started":1}
W20260620 00:18:28.081771 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:28.076670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57035 (local address 127.0.98.62:41543)
0620 00:18:28.076769 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:28.076772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:28.076940 (+ 168us) server_negotiation.cc:407] Connection header received
0620 00:18:28.077012 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:28.077013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:28.077030 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:28.077085 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:28.077459 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.078030 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.078845 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.079052 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.079110 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:28.081673 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:57035: BlockingRecv error: recv got EOF from 127.0.98.2:57035 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Jun 20 00:18:28 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:28 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:28.220278 9561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:28.215134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:55755)
0620 00:18:28.215327 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:28.215339 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:28.215406 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:28.215613 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:28.215614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:28.215879 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:28.215940 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:28.215942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.216609 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.216610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:28.217159 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:28.217161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.217484 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.217485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:28.217530 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:28.217971 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:28.217976 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:28.220183 (+ 2207us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":121,"threads_started":1}
W20260620 00:18:28.220321 9559 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:28.215282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55755 (local address 127.0.98.62:41543)
0620 00:18:28.215331 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:28.215333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:28.215398 (+ 65us) server_negotiation.cc:407] Connection header received
0620 00:18:28.215461 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:28.215462 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:28.215477 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:28.215574 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:28.216031 (+ 457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.216537 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.217234 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.217414 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.217468 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:28.220258 (+ 2790us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:55755: BlockingRecv error: recv got EOF from 127.0.98.3:55755 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Jun 20 00:18:28 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:28 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:28.950953 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:28.945214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:60597)
0620 00:18:28.945380 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:28.945394 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:28.945461 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:28.945688 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:28.945689 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:28.945854 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:28.945905 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:28.945907 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.946678 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.946679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:28.947275 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:28.947277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.947726 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.947727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:28.947782 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:28.948582 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:28.948587 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:28.950888 (+ 2301us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
W20260620 00:18:28.951196 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:28.945276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60597 (local address 127.0.98.62:41543)
0620 00:18:28.945433 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:28.945435 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:28.945447 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:18:28.945524 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:28.945525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:28.945534 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:28.945596 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:28.946006 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.946553 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.947391 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:28.947617 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:28.947691 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:28.951097 (+ 3406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:60597: BlockingRecv error: recv got EOF from 127.0.98.1:60597 (error 108)
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
Jun 20 00:18:29 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:29 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:29.087831 9564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:29.082218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:51015)
0620 00:18:29.082548 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:29.082562 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:29.082664 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:29.082835 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:29.082836 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:29.083023 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:29.083071 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:29.083073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.083780 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.083781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:29.084512 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:29.084515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.084907 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.084908 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:29.084967 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:29.085543 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:29.085548 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:29.087750 (+ 2202us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":211,"threads_started":1}
W20260620 00:18:29.087985 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:29.082452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51015 (local address 127.0.98.62:41543)
0620 00:18:29.082546 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:29.082548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:29.082647 (+ 99us) server_negotiation.cc:407] Connection header received
0620 00:18:29.082735 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:29.082736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:29.082750 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:29.082828 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:29.083154 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.083676 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.084604 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.084808 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.084903 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:29.087902 (+ 2999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51015: BlockingRecv error: recv got EOF from 127.0.98.2:51015 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Jun 20 00:18:29 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:29 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:29.225791 9565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:29.220958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:47329)
0620 00:18:29.221231 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:29.221250 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:29.221309 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:29.221451 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:29.221451 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:29.221624 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:29.221671 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:29.221672 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.222361 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.222362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:29.222907 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:29.222909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.223252 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.223253 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:29.223295 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:29.223736 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:29.223741 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:29.225732 (+ 1991us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":222,"threads_started":1}
W20260620 00:18:29.225960 9563 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:29.221064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47329 (local address 127.0.98.62:41543)
0620 00:18:29.221170 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:29.221173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:29.221313 (+ 140us) server_negotiation.cc:407] Connection header received
0620 00:18:29.221354 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:29.221355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:29.221367 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:29.221431 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:29.221733 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.222266 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.223002 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.223187 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.223245 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:29.225861 (+ 2616us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47329: BlockingRecv error: recv got EOF from 127.0.98.3:47329 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Jun 20 00:18:29 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:29 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:29.961704 9566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:29.951597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:46185)
0620 00:18:29.951749 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:29.951770 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:29.951846 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:29.952182 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:29.952184 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:29.952456 (+ 272us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:29.952521 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:29.952523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.953274 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.953276 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:29.953847 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:29.953848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.954201 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.954202 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:29.954252 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:29.959511 (+ 5259us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:29.959518 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:29.961638 (+ 2120us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":97,"threads_started":1}
W20260620 00:18:29.961942 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:29.951746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46185 (local address 127.0.98.62:41543)
0620 00:18:29.951891 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:29.951894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:29.951912 (+ 18us) server_negotiation.cc:407] Connection header received
0620 00:18:29.951952 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:29.951953 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:29.951964 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:29.952097 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:29.952630 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.953198 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.953927 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:29.954126 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:29.954183 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:29.961867 (+ 7684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46185: BlockingRecv error: recv got EOF from 127.0.98.1:46185 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":85,"threads_started":1}
Jun 20 00:18:30 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:30 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:30.093473 9568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:30.088475 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:56839)
0620 00:18:30.088639 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:30.088653 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:30.088711 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:30.088846 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:30.088847 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:30.089024 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:30.089073 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:30.089075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.089735 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.089736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:30.090287 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:30.090288 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.090699 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.090700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:30.090742 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:30.091295 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:30.091300 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:30.093388 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":73,"threads_started":1}
W20260620 00:18:30.093568 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:30.088618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56839 (local address 127.0.98.62:41543)
0620 00:18:30.088685 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:30.088687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:30.088711 (+ 24us) server_negotiation.cc:407] Connection header received
0620 00:18:30.088768 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:30.088769 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:30.088780 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:30.088824 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:30.089138 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.089663 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.090384 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.090618 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.090683 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:30.093478 (+ 2795us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:56839: BlockingRecv error: recv got EOF from 127.0.98.2:56839 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Jun 20 00:18:30 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:30 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:30.231427 9569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:30.226444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:46399)
0620 00:18:30.226649 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:30.226662 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:30.226745 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:30.226931 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:30.226932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:30.227118 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:30.227181 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:30.227182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.227938 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.227939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:30.228548 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:30.228550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.228904 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.228905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:30.228954 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:30.229504 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:30.229509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:30.231328 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":103,"threads_started":1}
W20260620 00:18:30.231510 9567 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:30.226583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46399 (local address 127.0.98.62:41543)
0620 00:18:30.226641 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:30.226644 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:30.226756 (+ 112us) server_negotiation.cc:407] Connection header received
0620 00:18:30.226793 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:30.226794 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:30.226808 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:30.226867 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:30.227257 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.227842 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.228625 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.228811 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.228866 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:30.231434 (+ 2568us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:46399: BlockingRecv error: recv got EOF from 127.0.98.3:46399 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Jun 20 00:18:30 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:30 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:30.967329 9570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:30.962331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:49873)
0620 00:18:30.962479 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:30.962490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:30.962545 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:30.962702 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:30.962703 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:30.962876 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:30.962929 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:30.962931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.963718 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.963718 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:30.964385 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:30.964388 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.964749 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.964750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:30.964794 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:30.965344 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:30.965350 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:30.967251 (+ 1901us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
W20260620 00:18:30.967522 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:30.962394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49873 (local address 127.0.98.62:41543)
0620 00:18:30.962532 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:30.962534 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:30.962544 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:30.962599 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:30.962599 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:30.962610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:30.962685 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:30.963060 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.963637 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.964468 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:30.964680 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:30.964738 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:30.967425 (+ 2687us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49873: BlockingRecv error: recv got EOF from 127.0.98.1:49873 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
Jun 20 00:18:31 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:31 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:31.099342 9572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:31.094091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:59385)
0620 00:18:31.094231 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:31.094242 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:31.094307 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:31.094532 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:31.094533 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:31.094752 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:31.094813 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:31.094815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.095693 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.095693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:31.096305 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:31.096307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.096666 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.096667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:31.096709 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:31.097194 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:31.097198 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:31.099241 (+ 2043us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":69,"threads_started":1}
W20260620 00:18:31.099406 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:31.094139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59385 (local address 127.0.98.62:41543)
0620 00:18:31.094184 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:31.094186 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:31.094299 (+ 113us) server_negotiation.cc:407] Connection header received
0620 00:18:31.094398 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:31.094417 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:31.094433 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:31.094492 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:31.095046 (+ 554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.095614 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.096400 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.096594 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.096658 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:31.099330 (+ 2672us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:59385: BlockingRecv error: recv got EOF from 127.0.98.2:59385 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 00:18:31 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:31 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:31.237340 9573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:31.232036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:41747)
0620 00:18:31.232372 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:31.232385 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:31.232493 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:31.232663 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:31.232664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:31.232859 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:31.232911 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:31.232913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.233606 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.233607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:31.234168 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:31.234170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.234563 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.234564 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:31.234622 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:31.235166 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:31.235170 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:31.237235 (+ 2065us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":222,"threads_started":1}
W20260620 00:18:31.237411 9571 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:31.232243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41747 (local address 127.0.98.62:41543)
0620 00:18:31.232358 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:31.232361 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:31.232471 (+ 110us) server_negotiation.cc:407] Connection header received
0620 00:18:31.232562 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:31.232563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:31.232584 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:31.232658 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:31.232999 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.233514 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.234268 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.234473 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.234562 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:31.237342 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:41747: BlockingRecv error: recv got EOF from 127.0.98.3:41747 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Jun 20 00:18:31 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:31 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:31.973167 9574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:31.967930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:57925)
0620 00:18:31.968133 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:31.968171 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:31.968241 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:31.968389 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:31.968390 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:31.968561 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:31.968612 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:31.968613 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.969255 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.969256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:31.969933 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:31.969936 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.970519 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.970520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:31.970590 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:31.971089 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:31.971094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:31.973101 (+ 2007us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":131,"threads_started":1}
W20260620 00:18:31.973340 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:31.968041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57925 (local address 127.0.98.62:41543)
0620 00:18:31.968188 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:31.968190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:31.968243 (+ 53us) server_negotiation.cc:407] Connection header received
0620 00:18:31.968293 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:31.968293 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:31.968304 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:31.968371 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:31.968678 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.969183 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.970035 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:31.970383 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:31.970451 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:31.973231 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57925: BlockingRecv error: recv got EOF from 127.0.98.1:57925 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":47,"threads_started":1}
Jun 20 00:18:32 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:32 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:32.105316 9576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:32.100026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:54879)
0620 00:18:32.100198 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:32.100210 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:32.100279 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:32.100475 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:32.100476 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:32.100662 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:32.100709 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:32.100711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.101388 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.101389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:32.101927 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:32.101929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.102304 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.102305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:32.102363 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:32.103128 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:32.103134 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:32.105244 (+ 2110us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":97,"threads_started":1}
W20260620 00:18:32.105394 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:32.100202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54879 (local address 127.0.98.62:41543)
0620 00:18:32.100261 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:32.100264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:32.100275 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:32.100347 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:32.100348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:32.100361 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:32.100462 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:32.100793 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.101298 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.102016 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.102216 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.102303 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:32.105347 (+ 3044us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54879: BlockingRecv error: recv got EOF from 127.0.98.2:54879 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Jun 20 00:18:32 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:32 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:32.242344 9577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:32.237850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:59485)
0620 00:18:32.238022 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:32.238032 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:32.238092 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:32.238251 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:32.238252 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:32.238427 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:32.238477 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:32.238478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.239153 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.239153 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:32.239672 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:32.239674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.240009 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.240010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:32.240049 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:32.240581 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:32.240586 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:32.242279 (+ 1693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
W20260620 00:18:32.242484 9575 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:32.237911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59485 (local address 127.0.98.62:41543)
0620 00:18:32.237985 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:32.237987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:32.238098 (+ 111us) server_negotiation.cc:407] Connection header received
0620 00:18:32.238137 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:32.238138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:32.238150 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:32.238201 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:32.238562 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.239072 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.239745 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.239945 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.239996 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:32.242397 (+ 2401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59485: BlockingRecv error: recv got EOF from 127.0.98.3:59485 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 00:18:32 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:32 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:32.978602 9578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:32.973725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:53119)
0620 00:18:32.973902 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:32.973913 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:32.973961 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:32.974154 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:32.974154 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:32.974347 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:32.974399 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:32.974402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.975060 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.975060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:32.975644 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:32.975646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.975983 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.975984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:32.976030 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:32.976567 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:32.976572 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:32.978492 (+ 1920us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
W20260620 00:18:32.978672 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:32.973880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53119 (local address 127.0.98.62:41543)
0620 00:18:32.974019 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:32.974021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:32.974032 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:32.974059 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:32.974060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:32.974069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:32.974134 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:32.974471 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.974982 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.975720 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:32.975919 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:32.975970 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:32.978601 (+ 2631us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:53119: BlockingRecv error: recv got EOF from 127.0.98.1:53119 (error 108)
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":90,"threads_started":1}
Jun 20 00:18:33 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:33 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:33.110574 9580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:33.105873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:54291)
0620 00:18:33.106031 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:33.106043 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:33.106115 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:33.106275 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:33.106276 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:33.106443 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:33.106491 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:33.106493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.107288 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.107289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:33.107822 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:33.107824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.108235 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.108236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:33.108296 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:33.108757 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:33.108762 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:33.110512 (+ 1750us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
W20260620 00:18:33.110773 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:33.105993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54291 (local address 127.0.98.62:41543)
0620 00:18:33.106064 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:33.106065 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:33.106099 (+ 34us) server_negotiation.cc:407] Connection header received
0620 00:18:33.106176 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:33.106176 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:33.106187 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:33.106256 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:33.106560 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.107203 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.107914 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.108100 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.108171 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:33.110649 (+ 2478us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54291: BlockingRecv error: recv got EOF from 127.0.98.2:54291 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Jun 20 00:18:33 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:33 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:33.248556 9581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:33.242933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:47787)
0620 00:18:33.243172 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:33.243197 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:33.243314 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:33.243544 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:33.243545 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:33.243753 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:33.243804 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:33.243805 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.244684 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.244686 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:33.245328 (+ 642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:33.245330 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.245750 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.245751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:33.245816 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:33.246360 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:33.246364 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:33.248448 (+ 2084us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":71,"threads_started":1}
W20260620 00:18:33.248621 9579 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:33.243131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47787 (local address 127.0.98.62:41543)
0620 00:18:33.243204 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:33.243207 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:33.243292 (+ 85us) server_negotiation.cc:407] Connection header received
0620 00:18:33.243407 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:33.243409 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:33.243423 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:33.243503 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:33.243897 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.244589 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.245427 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.245646 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.245748 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:33.248547 (+ 2799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:47787: BlockingRecv error: recv got EOF from 127.0.98.3:47787 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 00:18:33 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:33 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:33.984597 9582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:33.979161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:38633)
0620 00:18:33.979443 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:33.979454 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:33.979516 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:33.979682 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:33.979683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:33.979937 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:33.980013 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:33.980016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.980986 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.980988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:33.981558 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:33.981560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.981938 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.981939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:33.982000 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:33.982539 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:33.982549 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:33.984536 (+ 1987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":227,"threads_started":1}
W20260620 00:18:33.984747 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:33.979362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38633 (local address 127.0.98.62:41543)
0620 00:18:33.979489 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:33.979491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:33.979502 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:33.979583 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:33.979583 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:33.979595 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:33.979676 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:33.980103 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.980891 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.981643 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:33.981835 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:33.981896 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:33.984653 (+ 2757us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:38633: BlockingRecv error: recv got EOF from 127.0.98.1:38633 (error 108)
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":66,"threads_started":1}
Jun 20 00:18:34 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:34 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:34.116433 9584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:34.111228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:51305)
0620 00:18:34.111395 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:34.111407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:34.111473 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:34.111659 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:34.111660 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:34.111838 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:34.111889 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:34.111890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.112717 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.112718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:34.113260 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:34.113261 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.113652 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.113653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:34.113693 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:34.114190 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:34.114196 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:34.116309 (+ 2113us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":95,"threads_started":1}
W20260620 00:18:34.116449 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:34.111281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51305 (local address 127.0.98.62:41543)
0620 00:18:34.111352 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:34.111354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:34.111457 (+ 103us) server_negotiation.cc:407] Connection header received
0620 00:18:34.111542 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:34.111543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:34.111558 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:34.111602 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:34.112004 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.112608 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.113387 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.113582 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.113641 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:34.116395 (+ 2754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:51305: BlockingRecv error: recv got EOF from 127.0.98.2:51305 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Jun 20 00:18:34 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:34 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:34.253675 9585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:34.249156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:34275)
0620 00:18:34.249300 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:34.249312 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:34.249387 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:34.249515 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:34.249516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:34.249677 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:34.249726 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:34.249728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.250347 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.250347 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:34.250914 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:34.250916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.251228 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.251228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:34.251268 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:34.251746 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:34.251751 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:34.253616 (+ 1865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
W20260620 00:18:34.253885 9583 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:34.249240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34275 (local address 127.0.98.62:41543)
0620 00:18:34.249311 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:34.249313 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:34.249396 (+ 83us) server_negotiation.cc:407] Connection header received
0620 00:18:34.249425 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:34.249426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:34.249438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:34.249497 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:34.249796 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.250275 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.250991 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.251165 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.251215 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:34.253799 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:34275: BlockingRecv error: recv got EOF from 127.0.98.3:34275 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Jun 20 00:18:34 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:34 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:34.994453 9586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:34.985083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:50409)
0620 00:18:34.985294 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:34.985306 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:34.985363 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:34.985519 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:34.985520 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:34.985686 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:34.985746 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:34.985751 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.986491 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.986492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:34.987026 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:34.987027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.987428 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.987429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:34.987467 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:34.992263 (+ 4796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:34.992268 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:34.994358 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":107,"threads_started":1}
W20260620 00:18:34.994526 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:34.985206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50409 (local address 127.0.98.62:41543)
0620 00:18:34.985343 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:34.985345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:34.985355 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:34.985427 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:34.985428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:34.985438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:34.985500 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:34.985819 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.986399 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.987108 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:34.987352 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:34.987412 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:34.994459 (+ 7047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:50409: BlockingRecv error: recv got EOF from 127.0.98.1:50409 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
Jun 20 00:18:35 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:35 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:35.122577 9588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:35.116983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:55239)
0620 00:18:35.117260 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:35.117275 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:35.117385 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:35.117555 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:35.117556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:35.117804 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:35.117855 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:35.117857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.118634 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.118636 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:35.119186 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:35.119187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.119610 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.119617 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:35.119676 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:35.120260 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:35.120265 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:35.122509 (+ 2244us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":166,"threads_started":1}
W20260620 00:18:35.122758 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:35.117133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55239 (local address 127.0.98.62:41543)
0620 00:18:35.117239 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:35.117242 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:35.117362 (+ 120us) server_negotiation.cc:407] Connection header received
0620 00:18:35.117463 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:35.117464 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:35.117479 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:35.117551 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:35.117962 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.118523 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.119299 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.119513 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.119609 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:35.122683 (+ 3074us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:55239: BlockingRecv error: recv got EOF from 127.0.98.2:55239 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Jun 20 00:18:35 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:35 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:35.259421 9589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:35.254284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:56799)
0620 00:18:35.254465 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:35.254475 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:35.254530 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:35.254697 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:35.254697 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:35.254861 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:35.254911 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:35.254914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.255612 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.255613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:35.256173 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:35.256175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.256614 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.256615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:35.256672 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:35.257244 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:35.257255 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:35.259344 (+ 2089us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":77,"threads_started":1}
W20260620 00:18:35.259588 9587 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:35.254437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56799 (local address 127.0.98.62:41543)
0620 00:18:35.254509 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:35.254511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:35.254525 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:18:35.254593 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:35.254593 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:35.254605 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:35.254668 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:35.255001 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.255512 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.256248 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.256520 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.256578 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:35.259504 (+ 2926us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56799: BlockingRecv error: recv got EOF from 127.0.98.3:56799 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Jun 20 00:18:35 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:35 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:36.000192 9590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:35.995025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:59265)
0620 00:18:35.995210 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:35.995222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:35.995288 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:35.995525 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:35.995527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:35.995822 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:35.995894 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:35.995897 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.996805 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.996805 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:35.997336 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:35.997338 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.997700 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.997700 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:35.997738 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:35.998276 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:35.998281 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:36.000088 (+ 1807us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":91,"threads_started":1}
W20260620 00:18:36.000380 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:35.995155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59265 (local address 127.0.98.62:41543)
0620 00:18:35.995273 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:35.995275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:35.995285 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:18:35.995397 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:35.995398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:35.995411 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:35.995502 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:35.996023 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.996711 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.997430 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:35.997631 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:35.997691 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:36.000294 (+ 2603us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:59265: BlockingRecv error: recv got EOF from 127.0.98.1:59265 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":53,"threads_started":1}
Jun 20 00:18:36 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:36 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:36.127899 9592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:36.123148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:43633)
0620 00:18:36.123397 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:36.123409 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:36.123468 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:36.123654 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:36.123659 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:36.123855 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:36.123903 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:36.123904 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:36.124678 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:36.124679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:36.125210 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:36.125212 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:36.125555 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:36.125556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:36.125597 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:36.126068 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:36.126073 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:36.127838 (+ 1765us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":164,"threads_started":1}
W20260620 00:18:36.128029 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:36.123347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43633 (local address 127.0.98.62:41543)
0620 00:18:36.123443 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:36.123445 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:36.123458 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:18:36.123524 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:36.123525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:36.123537 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:36.123593 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:36.124010 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:36.124594 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:36.125295 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:36.125487 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:36.125544 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:36.127946 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43633: BlockingRecv error: recv got EOF from 127.0.98.2:43633 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Jun 20 00:18:36 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:36 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:36.265296 9593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:36.259946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:40983)
0620 00:18:36.260194 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:36.260207 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:36.260306 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:36.260472 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:36.260473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:36.260640 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:36.260689 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:36.260691 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:36.261507 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:36.261508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:36.262218 (+ 710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:36.262221 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:36.262600 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:36.262601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:36.262641 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:36.263118 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:36.263123 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:36.265185 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":114,"threads_started":1}
W20260620 00:18:36.265336 9591 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:36.260059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40983 (local address 127.0.98.62:41543)
0620 00:18:36.260177 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:36.260179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:36.260312 (+ 133us) server_negotiation.cc:407] Connection header received
0620 00:18:36.260380 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:36.260381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:36.260394 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:36.260447 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:36.260804 (+ 357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:36.261429 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:36.262319 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:36.262526 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:36.262595 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:36.265269 (+ 2674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40983: BlockingRecv error: recv got EOF from 127.0.98.3:40983 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Jun 20 00:18:37 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:37 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:37.005785 9594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:37.000768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:44899)
0620 00:18:37.000965 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:37.000977 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:37.001042 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:37.001232 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:37.001233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:37.001404 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:37.001455 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:37.001456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.002100 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.002101 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:37.002736 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:37.002739 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.003107 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.003108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:37.003161 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:37.003813 (+ 652us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:37.003818 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:37.005715 (+ 1897us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":98,"threads_started":1}
W20260620 00:18:37.005906 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:37.000869 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:44899 (local address 127.0.98.62:41543)
0620 00:18:37.000988 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:37.000992 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:37.001024 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:18:37.001102 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:37.001103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:37.001113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:37.001180 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:37.001521 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.002023 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.002821 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.003021 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.003077 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:37.005818 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:44899: BlockingRecv error: recv got EOF from 127.0.98.1:44899 (error 108)
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":50,"threads_started":1}
Jun 20 00:18:37 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:37 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:37.134099 9596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:37.128520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:44417)
0620 00:18:37.128825 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:37.128840 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:37.128924 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:37.129081 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:37.129082 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:37.129277 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:37.129327 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:37.129329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.130041 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.130042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:37.130617 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:37.130619 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.131021 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.131022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:37.131079 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:37.131635 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:37.131641 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:37.133990 (+ 2349us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":283,"thread_start_us":108,"threads_started":1}
W20260620 00:18:37.134158 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:37.128653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44417 (local address 127.0.98.62:41543)
0620 00:18:37.128830 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:37.128833 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:37.128900 (+ 67us) server_negotiation.cc:407] Connection header received
0620 00:18:37.128986 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:37.128987 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:37.129000 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:37.129075 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:37.129442 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.129948 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.130718 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.130937 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.131021 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:37.134093 (+ 3072us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44417: BlockingRecv error: recv got EOF from 127.0.98.2:44417 (error 108)
Metrics: {"server-negotiator.queue_time_us":144}
Jun 20 00:18:37 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:37 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:37.270284 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:37.265803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:40485)
0620 00:18:37.265976 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:37.265986 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:37.266062 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:37.266192 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:37.266193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:37.266378 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:37.266425 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:37.266427 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.267064 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.267064 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:37.267587 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:37.267589 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.267937 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.267937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:37.267976 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:37.268405 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:37.268409 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:37.270224 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":115,"threads_started":1}
W20260620 00:18:37.270524 9595 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:37.265864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40485 (local address 127.0.98.62:41543)
0620 00:18:37.265932 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:37.265934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:37.266047 (+ 113us) server_negotiation.cc:407] Connection header received
0620 00:18:37.266122 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:37.266122 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:37.266134 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:37.266184 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:37.266496 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.266996 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.267670 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:37.267873 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:37.267926 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:37.270436 (+ 2510us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40485: BlockingRecv error: recv got EOF from 127.0.98.3:40485 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Jun 20 00:18:38 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:38 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:38.011904 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:38.006424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:37889)
0620 00:18:38.006577 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:38.006592 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:38.006660 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:38.006927 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:38.006928 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:38.007108 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:38.007158 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:38.007160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.008307 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.008309 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:38.008859 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:38.008861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.009247 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.009249 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:38.009304 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:38.009819 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:38.009823 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:38.011837 (+ 2014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
W20260620 00:18:38.012091 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:38.006610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37889 (local address 127.0.98.62:41543)
0620 00:18:38.006754 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:38.006755 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:38.006764 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:38.006789 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:38.006789 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:38.006800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:38.006874 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:38.007398 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.008227 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.008960 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.009161 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.009234 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:38.012002 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37889: BlockingRecv error: recv got EOF from 127.0.98.1:37889 (error 108)
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":91,"threads_started":1}
Jun 20 00:18:38 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:38 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:38.139714 9600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:38.134773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:52023)
0620 00:18:38.134935 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:38.134953 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:38.135049 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:38.135204 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:38.135205 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:38.135419 (+ 214us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:38.135468 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:38.135470 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.136240 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.136241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:38.136788 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:38.136790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.137198 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.137199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:38.137264 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:38.137800 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:38.137805 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:38.139609 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":80,"threads_started":1}
W20260620 00:18:38.139767 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:38.134974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52023 (local address 127.0.98.62:41543)
0620 00:18:38.135056 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:38.135058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:38.135086 (+ 28us) server_negotiation.cc:407] Connection header received
0620 00:18:38.135118 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:38.135119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:38.135130 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:38.135181 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:38.135562 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.136091 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.136888 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.137106 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.137196 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:38.139703 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:52023: BlockingRecv error: recv got EOF from 127.0.98.2:52023 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Jun 20 00:18:38 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:38 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:38.276547 9601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:38.270867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:45785)
0620 00:18:38.271087 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:38.271098 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:38.271191 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:38.271400 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:38.271401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:38.271609 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:38.271681 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:38.271683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.272481 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.272483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:38.273211 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:38.273261 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.273674 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.273676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:38.273730 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:38.274345 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:38.274350 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:38.276477 (+ 2127us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":135,"threads_started":1}
W20260620 00:18:38.276671 9599 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:38.270984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45785 (local address 127.0.98.62:41543)
0620 00:18:38.271104 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:38.271106 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:38.271191 (+ 85us) server_negotiation.cc:407] Connection header received
0620 00:18:38.271285 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:38.271286 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:38.271301 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:38.271368 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:38.271800 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.272378 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.273384 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:38.273582 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:38.273644 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:38.276578 (+ 2934us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45785: BlockingRecv error: recv got EOF from 127.0.98.3:45785 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Jun 20 00:18:39 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:39 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:39.017381 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:39.012497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:36207)
0620 00:18:39.012628 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:39.012640 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:39.012703 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:39.012874 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:39.012875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:39.013035 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:39.013086 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:39.013088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.013780 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.013781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:39.014313 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:39.014315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.014655 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.014656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:39.014694 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:39.015293 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:39.015299 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:39.017312 (+ 2013us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":70,"threads_started":1}
W20260620 00:18:39.017529 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:39.012567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36207 (local address 127.0.98.62:41543)
0620 00:18:39.012715 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:39.012717 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:39.012763 (+ 46us) server_negotiation.cc:407] Connection header received
0620 00:18:39.012789 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:39.012790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:39.012801 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:39.012858 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:39.013155 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.013688 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.014396 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.014589 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.014642 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:39.017438 (+ 2796us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36207: BlockingRecv error: recv got EOF from 127.0.98.1:36207 (error 108)
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
Jun 20 00:18:39 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:39 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:39.144948 9604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:39.140307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:58019)
0620 00:18:39.140482 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:39.140494 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:39.140559 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:39.140684 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:39.140685 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:39.140862 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:39.140911 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:39.140913 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.141597 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.141598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:39.142126 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:39.142127 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.142453 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.142454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:39.142493 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:39.142951 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:39.142955 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:39.144876 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":130,"threads_started":1}
W20260620 00:18:39.145061 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:39.140388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58019 (local address 127.0.98.62:41543)
0620 00:18:39.140464 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:39.140467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:39.140559 (+ 92us) server_negotiation.cc:407] Connection header received
0620 00:18:39.140620 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:39.140621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:39.140633 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:39.140674 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:39.140988 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.141510 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.142203 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.142390 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.142440 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:39.144986 (+ 2546us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58019: BlockingRecv error: recv got EOF from 127.0.98.2:58019 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Jun 20 00:18:39 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:39 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:39.282812 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:39.277250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.3:50649)
0620 00:18:39.277454 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:39.277464 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:39.277577 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:39.277768 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:39.277769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:39.277930 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:39.278007 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:39.278009 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.278785 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.278787 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:39.279682 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:39.279685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.280058 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.280060 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:39.280108 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:39.280812 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:39.280817 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:39.282728 (+ 1911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":117,"threads_started":1}
W20260620 00:18:39.282886 9603 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:39.277345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50649 (local address 127.0.98.62:41543)
0620 00:18:39.277430 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:39.277432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:39.277558 (+ 126us) server_negotiation.cc:407] Connection header received
0620 00:18:39.277648 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:39.277649 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:39.277663 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:39.277748 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:39.278085 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.278602 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.279766 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:39.279966 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:39.280028 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:39.282836 (+ 2808us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50649: BlockingRecv error: recv got EOF from 127.0.98.3:50649 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Jun 20 00:18:40 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:40 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:40.026480 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.017965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.1:49609)
0620 00:18:40.018139 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:40.018150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:40.018199 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:40.018403 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:40.018403 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:40.018562 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:40.018609 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.018610 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.019270 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.019271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.019888 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.019892 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.020409 (+ 517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.020410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.020473 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.024490 (+ 4017us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:40.024496 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:40.026376 (+ 1880us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":111,"threads_started":1}
W20260620 00:18:40.026556 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:40.018055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49609 (local address 127.0.98.62:41543)
0620 00:18:40.018206 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:40.018207 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:40.018293 (+ 86us) server_negotiation.cc:407] Connection header received
0620 00:18:40.018318 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:40.018319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:40.018329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:40.018383 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:40.018675 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.019175 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.019989 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.020310 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.020395 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.026475 (+ 6080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:49609: BlockingRecv error: recv got EOF from 127.0.98.1:49609 (error 108)
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
Jun 20 00:18:40 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Jun 20 00:18:40 dist-test-slave-r81b krb5kdc[8549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260620 00:18:40.150213 9608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.145540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41543 (local address 127.0.98.2:41885)
0620 00:18:40.145699 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:40.145714 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:40.145771 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:40.145917 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:40.145918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:40.146089 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:40.146138 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.146139 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.146796 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.146797 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.147318 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.147319 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.147659 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.147659 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.147699 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.148217 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:40.148225 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:40.150142 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:41543: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
W20260620 00:18:40.150305 9607 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:40.145609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41885 (local address 127.0.98.62:41543)
0620 00:18:40.145685 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:40.145687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:40.145810 (+ 123us) server_negotiation.cc:407] Connection header received
0620 00:18:40.145837 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:40.145838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:40.145849 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:40.145898 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:40.146209 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.146708 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.147402 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.147593 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.147645 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.150257 (+ 2612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41885: BlockingRecv error: recv got EOF from 127.0.98.2:41885 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260620 00:18:40.195839 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 9132
I20260620 00:18:40.200531 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 9284
I20260620 00:18:40.204954 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 9416
I20260620 00:18:40.209278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 9062
2026-06-20T00:18:40Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17526 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-06-20T00:18:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:40Z Disabled control of system clock
I20260620 00:18:40.237639 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45027
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45027
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260620 00:18:40.318900 9612 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:40.319126 9612 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:40.319146 9612 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:40.320365 9612 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:40.320407 9612 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:40.320420 9612 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:40.320433 9612 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:40.320446 9612 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:40.321735 9612 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45027
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45027
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9612
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:40.321943 9612 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:40.322233 9612 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:40.327378 9618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:40.327364 9620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:40.327400 9612 server_base.cc:1034] running on GCE node
W20260620 00:18:40.327358 9617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:40.328035 9612 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:40.328444 9612 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:40.329631 9612 hybrid_clock.cc:648] HybridClock initialized: now 1781914720329622 us; error 61 us; skew 500 ppm
I20260620 00:18:40.331045 9612 webserver.cc:460] Webserver started at http://127.0.98.62:41167/ using document root <none> and password file <none>
I20260620 00:18:40.331300 9612 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:40.331380 9612 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:40.331528 9612 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:40.332691 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "219b205ab1ec4ed08077dc5ef6904943"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "f6fd77cbf2c794dd89921e97534b4ec7"
server_key_iv: "07de51f51cfee56126fab4cc6536b984"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.333062 9612 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "219b205ab1ec4ed08077dc5ef6904943"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "f6fd77cbf2c794dd89921e97534b4ec7"
server_key_iv: "07de51f51cfee56126fab4cc6536b984"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.334442 9612 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:40.335201 9626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.335430 9612 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:40.335523 9612 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "219b205ab1ec4ed08077dc5ef6904943"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "f6fd77cbf2c794dd89921e97534b4ec7"
server_key_iv: "07de51f51cfee56126fab4cc6536b984"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.335582 9612 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:40.351450 9612 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:40.351743 9612 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:40.351862 9612 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:40.356531 9612 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45027
I20260620 00:18:40.356630 9678 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45027 every 8 connection(s)
I20260620 00:18:40.357051 9612 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:40.358027 9679 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:40.360359 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943: Bootstrap starting.
I20260620 00:18:40.360950 9679 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:40.361248 9679 log.cc:824] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:40.361943 9679 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943: No bootstrap required, opened a new log
I20260620 00:18:40.362201 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 9612
I20260620 00:18:40.362403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:40.362600 392 external_mini_cluster.cc:1428] Setting key dcd75de1d8edbef7a3b834bd796164ed
I20260620 00:18:40.364354 9679 raft_consensus.cc:348] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219b205ab1ec4ed08077dc5ef6904943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } }
I20260620 00:18:40.364532 9679 raft_consensus.cc:374] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:40.364573 9679 raft_consensus.cc:729] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 219b205ab1ec4ed08077dc5ef6904943, State: Initialized, Role: FOLLOWER
I20260620 00:18:40.364717 9679 consensus_queue.cc:260] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219b205ab1ec4ed08077dc5ef6904943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } }
I20260620 00:18:40.364799 9679 raft_consensus.cc:388] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:40.364838 9679 raft_consensus.cc:482] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:40.364892 9679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:40.365612 9679 raft_consensus.cc:504] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219b205ab1ec4ed08077dc5ef6904943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } }
I20260620 00:18:40.365721 9679 leader_election.cc:302] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 219b205ab1ec4ed08077dc5ef6904943; no voters:
I20260620 00:18:40.365860 9679 leader_election.cc:288] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:40.365896 9684 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:40.366051 9684 raft_consensus.cc:686] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [term 1 LEADER]: Becoming Leader. State: Replica: 219b205ab1ec4ed08077dc5ef6904943, State: Running, Role: LEADER
I20260620 00:18:40.366137 9679 sys_catalog.cc:564] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:40.366184 9684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219b205ab1ec4ed08077dc5ef6904943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } }
I20260620 00:18:40.366668 9685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "219b205ab1ec4ed08077dc5ef6904943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219b205ab1ec4ed08077dc5ef6904943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } } }
I20260620 00:18:40.366750 9685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:40.366969 9686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 219b205ab1ec4ed08077dc5ef6904943. Latest consensus state: current_term: 1 leader_uuid: "219b205ab1ec4ed08077dc5ef6904943" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "219b205ab1ec4ed08077dc5ef6904943" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45027 } } }
I20260620 00:18:40.367080 9686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:40.367246 9693 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:40.367825 9693 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:40.369590 9693 catalog_manager.cc:1347] Generated new cluster ID: a122a5870d684d88ab9bc5f5149833dc
I20260620 00:18:40.369645 9693 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:40.369831 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.363138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34260 (local address 127.0.98.62:45027)
0620 00:18:40.363382 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:40.363392 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:40.363425 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:18:40.363912 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:40.363921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:40.364000 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:40.364180 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:40.364391 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.366120 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.367382 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.368051 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.368203 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.368971 (+ 768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:40.368975 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:40.368976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:40.369181 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:40.369223 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:40.369237 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:40.369306 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:40.369425 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 00:18:40.369610 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":99,"threads_started":1}
I20260620 00:18:40.376392 9693 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:40.376935 9693 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:40.383438 9693 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 219b205ab1ec4ed08077dc5ef6904943: Generated new TSK 0
I20260620 00:18:40.383685 9693 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:18:40.390945 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45027
--builtin_ntp_servers=127.0.98.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260620 00:18:40.493234 9703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:40.493482 9703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:40.493521 9703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:40.494731 9703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:40.494800 9703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:40.494843 9703 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:40.496381 9703 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:40.496672 9703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:40.496980 9703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:40.502012 9711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:40.502017 9709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:40.502267 9703 server_base.cc:1034] running on GCE node
W20260620 00:18:40.502025 9708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:40.502552 9703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:40.502863 9703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:40.504045 9703 hybrid_clock.cc:648] HybridClock initialized: now 1781914720504021 us; error 37 us; skew 500 ppm
I20260620 00:18:40.505466 9703 webserver.cc:460] Webserver started at http://127.0.98.1:40381/ using document root <none> and password file <none>
I20260620 00:18:40.505730 9703 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:40.505779 9703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:40.505975 9703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:40.507076 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "f58a4c3ca81949e18edb71176b500e4c"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "35e07d793eb553c059160137e1bd0aff"
server_key_iv: "ac0023ad79693b3b2fc7046eaf9fffaa"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.507437 9703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f58a4c3ca81949e18edb71176b500e4c"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "35e07d793eb553c059160137e1bd0aff"
server_key_iv: "ac0023ad79693b3b2fc7046eaf9fffaa"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.508893 9703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:40.509684 9717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.509845 9703 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:18:40.509958 9703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "f58a4c3ca81949e18edb71176b500e4c"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "35e07d793eb553c059160137e1bd0aff"
server_key_iv: "ac0023ad79693b3b2fc7046eaf9fffaa"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.510037 9703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:40.525646 9703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:40.525996 9703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:40.526154 9703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:40.526399 9703 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:40.526731 9703 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:40.526765 9703 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.526788 9703 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:40.526800 9703 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.533262 9703 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39613
I20260620 00:18:40.533349 9830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39613 every 8 connection(s)
I20260620 00:18:40.533682 9703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:18:40.536650 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 9703
I20260620 00:18:40.536760 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:40.536943 392 external_mini_cluster.cc:1428] Setting key 1fca5753149f79ea733c2b1dcb9720d5
I20260620 00:18:40.537921 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45027
--builtin_ntp_servers=127.0.98.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 00:18:40.539669 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.534474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39583 (local address 127.0.98.62:45027)
0620 00:18:40.534554 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:40.534557 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:40.534728 (+ 171us) server_negotiation.cc:407] Connection header received
0620 00:18:40.535227 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:40.535228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:40.535238 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:40.535315 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:40.535944 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.536508 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.537985 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.538336 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.538411 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.539426 (+ 1015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:40.539427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:40.539428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:40.539432 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:40.539445 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:40.539449 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:40.539513 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:40.539578 (+ 65us) server_negotiation.cc:299] Negotiation successful
0620 00:18:40.539611 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:18:40.539961 9833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.534409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45027 (local address 127.0.98.1:39583)
0620 00:18:40.534607 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:40.534642 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:40.535143 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:40.535377 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:40.535380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:40.535390 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:40.535872 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.535876 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.536621 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.536622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.537898 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.537900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.538452 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.538453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.538592 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.539191 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:40.539191 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:40.539300 (+ 109us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:40.539336 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:40.539497 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:40.539501 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:40.539504 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:40.539583 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 00:18:40.539701 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":97,"threads_started":1}
I20260620 00:18:40.541306 9831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45027
I20260620 00:18:40.541426 9831 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:40.541625 9831 heartbeater.cc:511] Master 127.0.98.62:45027 requested a full tablet report, sending...
I20260620 00:18:40.542251 9643 ts_manager.cc:194] Registered new tserver with Master: f58a4c3ca81949e18edb71176b500e4c (127.0.98.1:39613)
I20260620 00:18:40.543268 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:39583
W20260620 00:18:40.621687 9834 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:40.621896 9834 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:40.621917 9834 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:40.623000 9834 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:40.623072 9834 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:40.623116 9834 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:40.624432 9834 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9834
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:40.624619 9834 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:40.624857 9834 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:40.629977 9840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:40.629984 9842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:40.629984 9839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:40.630394 9834 server_base.cc:1034] running on GCE node
I20260620 00:18:40.630589 9834 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:40.630798 9834 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:40.631953 9834 hybrid_clock.cc:648] HybridClock initialized: now 1781914720631934 us; error 32 us; skew 500 ppm
I20260620 00:18:40.633410 9834 webserver.cc:460] Webserver started at http://127.0.98.2:34919/ using document root <none> and password file <none>
I20260620 00:18:40.633625 9834 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:40.633666 9834 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:40.633755 9834 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:40.634778 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "eb35c60f71fd4192958e5e9c5d76f1a9"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "2757a045a174ad8f2e3dc6fc0e5e053c"
server_key_iv: "317901d3177015300f142eb3f8a6a8cd"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.635077 9834 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "eb35c60f71fd4192958e5e9c5d76f1a9"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "2757a045a174ad8f2e3dc6fc0e5e053c"
server_key_iv: "317901d3177015300f142eb3f8a6a8cd"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.636397 9834 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:40.637171 9848 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.637353 9834 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:40.637483 9834 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "eb35c60f71fd4192958e5e9c5d76f1a9"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "2757a045a174ad8f2e3dc6fc0e5e053c"
server_key_iv: "317901d3177015300f142eb3f8a6a8cd"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.637600 9834 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:40.659713 9834 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:40.660077 9834 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:40.660277 9834 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:40.660543 9834 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:40.660904 9834 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:40.660964 9834 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.661023 9834 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:40.661059 9834 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.667631 9834 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35583
I20260620 00:18:40.667666 9961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35583 every 8 connection(s)
I20260620 00:18:40.668046 9834 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:40.673957 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 9834
I20260620 00:18:40.673954 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.668928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42203 (local address 127.0.98.62:45027)
0620 00:18:40.668986 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:40.668989 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:40.669222 (+ 233us) server_negotiation.cc:407] Connection header received
0620 00:18:40.669812 (+ 590us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:40.669814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:40.669827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:40.669889 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:40.670689 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.671240 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.672397 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.672609 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.672695 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.673625 (+ 930us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:40.673627 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:40.673627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:40.673632 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:40.673654 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:40.673658 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:40.673718 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:40.673829 (+ 111us) server_negotiation.cc:299] Negotiation successful
0620 00:18:40.673859 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:18:40.674079 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:40.674185 9964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.668895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45027 (local address 127.0.98.2:42203)
0620 00:18:40.669080 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:40.669121 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:40.669690 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:40.669977 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:40.669981 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:40.669992 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:40.670576 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.670581 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.671368 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.671370 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.672301 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.672304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.672703 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.672704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.672865 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.673446 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:40.673446 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:40.673478 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:40.673503 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:40.673719 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:40.673726 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:40.673729 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:40.673832 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 00:18:40.673942 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":66,"threads_started":1}
I20260620 00:18:40.674283 392 external_mini_cluster.cc:1428] Setting key 0d7d8a6f8b5e87a50417ecd624742f16
I20260620 00:18:40.674700 9962 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45027
I20260620 00:18:40.674804 9962 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:40.675014 9962 heartbeater.cc:511] Master 127.0.98.62:45027 requested a full tablet report, sending...
I20260620 00:18:40.675428 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45027
--builtin_ntp_servers=127.0.98.20:35843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260620 00:18:40.675469 9643 ts_manager.cc:194] Registered new tserver with Master: eb35c60f71fd4192958e5e9c5d76f1a9 (127.0.98.2:35583)
I20260620 00:18:40.675966 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:42203
W20260620 00:18:40.758126 9965 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:40.758359 9965 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:40.758383 9965 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:40.759570 9965 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:40.759615 9965 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:40.759640 9965 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:40.761035 9965 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45027
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9965
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:40.761269 9965 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:40.761584 9965 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:40.766532 9973 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:40.766585 9970 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:40.766538 9971 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:40.766805 9965 server_base.cc:1034] running on GCE node
I20260620 00:18:40.767000 9965 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:40.767215 9965 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:40.768405 9965 hybrid_clock.cc:648] HybridClock initialized: now 1781914720768388 us; error 25 us; skew 500 ppm
I20260620 00:18:40.769753 9965 webserver.cc:460] Webserver started at http://127.0.98.3:40661/ using document root <none> and password file <none>
I20260620 00:18:40.770006 9965 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:40.770079 9965 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:40.770228 9965 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:40.771339 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "8c327869d533400999e7f6b677a6148d"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "7580a85a1107a31a86ea04037b79e2a7"
server_key_iv: "bf25dad647ae467c69e5c70503a1612c"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.771708 9965 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8c327869d533400999e7f6b677a6148d"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "7580a85a1107a31a86ea04037b79e2a7"
server_key_iv: "bf25dad647ae467c69e5c70503a1612c"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.773129 9965 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:40.773836 9979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.774001 9965 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260620 00:18:40.774108 9965 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "8c327869d533400999e7f6b677a6148d"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "7580a85a1107a31a86ea04037b79e2a7"
server_key_iv: "bf25dad647ae467c69e5c70503a1612c"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.774183 9965 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:40.790216 9965 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:40.790572 9965 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:40.790737 9965 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:40.791004 9965 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:40.791340 9965 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:40.791399 9965 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.791441 9965 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:40.791483 9965 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.797353 9965 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44947
I20260620 00:18:40.797407 10092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44947 every 8 connection(s)
I20260620 00:18:40.797739 9965 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:40.800812 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 9965
I20260620 00:18:40.800908 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:40.801077 392 external_mini_cluster.cc:1428] Setting key 5faa82703b2d8930acc02e295153c88d
I20260620 00:18:40.803236 9683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.798481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37987 (local address 127.0.98.62:45027)
0620 00:18:40.798538 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:40.798540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:40.798764 (+ 224us) server_negotiation.cc:407] Connection header received
0620 00:18:40.799282 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:40.799283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:40.799293 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:40.799361 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:40.800001 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.800508 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.802029 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.802212 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.802266 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.803012 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:40.803013 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:40.803014 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:40.803017 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:40.803031 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:40.803035 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:40.803078 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:40.803158 (+ 80us) server_negotiation.cc:299] Negotiation successful
0620 00:18:40.803179 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:18:40.803457 10095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.798472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45027 (local address 127.0.98.3:37987)
0620 00:18:40.798644 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:40.798678 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:40.799172 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:40.799418 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:40.799422 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:40.799430 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:40.799927 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.799931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.800592 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.800593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.801952 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:40.801955 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:40.802288 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:40.802288 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:40.802398 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:40.802885 (+ 487us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:40.802885 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:40.802913 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:40.802936 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:40.803091 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:40.803094 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:40.803097 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:40.803149 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:18:40.803223 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260620 00:18:40.804015 10093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45027
I20260620 00:18:40.804091 10093 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:40.804294 10093 heartbeater.cc:511] Master 127.0.98.62:45027 requested a full tablet report, sending...
I20260620 00:18:40.804622 9643 ts_manager.cc:194] Registered new tserver with Master: 8c327869d533400999e7f6b677a6148d (127.0.98.3:44947)
I20260620 00:18:40.804986 9643 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:37987
I20260620 00:18:40.813022 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:18:40.815294 9683 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:40.814618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34276 (local address 127.0.98.62:45027)
0620 00:18:40.814721 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:40.814723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:40.814801 (+ 78us) server_negotiation.cc:407] Connection header received
0620 00:18:40.814855 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:40.814855 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:40.814864 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:40.814914 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:40.815224 (+ 310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34276: BlockingRecv error: recv got EOF from 127.0.0.1:34276 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260620 00:18:40.815363 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 9703
I20260620 00:18:40.820061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 9834
I20260620 00:18:40.824555 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 9965
I20260620 00:18:40.828672 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 9612
2026-06-20T00:18:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (618 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-06-20T00:18:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:40Z Disabled control of system clock
I20260620 00:18:40.855654 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34489
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:18:40.941125 10104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:40.941337 10104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:40.941360 10104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:40.942523 10104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:40.942566 10104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:40.942579 10104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:40.942592 10104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:40.942643 10104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:40.944053 10104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34489
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:40.944348 10104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:40.944649 10104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:18:40.949905 10104 server_base.cc:1034] running on GCE node
W20260620 00:18:40.949879 10112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:40.949855 10110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:40.950012 10109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:40.950459 10104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:40.950858 10104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:40.952061 10104 hybrid_clock.cc:648] HybridClock initialized: now 1781914720952018 us; error 46 us; skew 500 ppm
I20260620 00:18:40.953459 10104 webserver.cc:460] Webserver started at http://127.0.98.62:35467/ using document root <none> and password file <none>
I20260620 00:18:40.953769 10104 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:40.953850 10104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:40.953994 10104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:40.955158 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "0dec00cfa872446b9bc1cb760c0ef221"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "01eb98b97b1ca89b02907aa2bc0f87fe"
server_key_iv: "5360cbccc7279c35b1db5a3acb6eef8d"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.955572 10104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "0dec00cfa872446b9bc1cb760c0ef221"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "01eb98b97b1ca89b02907aa2bc0f87fe"
server_key_iv: "5360cbccc7279c35b1db5a3acb6eef8d"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.957187 10104 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:40.958034 10118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:40.958220 10104 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:40.958338 10104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "0dec00cfa872446b9bc1cb760c0ef221"
format_stamp: "Formatted at 2026-06-20 00:18:40 on dist-test-slave-r81b"
server_key: "01eb98b97b1ca89b02907aa2bc0f87fe"
server_key_iv: "5360cbccc7279c35b1db5a3acb6eef8d"
server_key_version: "encryptionkey@0"
I20260620 00:18:40.958421 10104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:40.961999 10104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:40.962302 10104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:40.962445 10104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:40.966447 10104 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34489
I20260620 00:18:40.966477 10170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34489 every 8 connection(s)
I20260620 00:18:40.966820 10104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:40.967514 10171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:40.970312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 10104
I20260620 00:18:40.970422 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:40.970638 392 external_mini_cluster.cc:1428] Setting key 2bc1b293513682b128ba50889625add4
I20260620 00:18:40.971167 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221: Bootstrap starting.
I20260620 00:18:40.971868 10171 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:40.972231 10171 log.cc:824] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:40.973368 10171 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221: No bootstrap required, opened a new log
I20260620 00:18:40.973701 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:40.971373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42430 (local address 127.0.98.62:34489)
0620 00:18:40.971577 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:40.971585 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:40.971617 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:18:40.972120 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:40.972128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:40.972294 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:40.972496 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:40.973192 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:40.973196 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:40.973196 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:40.973200 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:40.973241 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:40.973252 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:40.973358 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:40.973477 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 00:18:40.973542 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260620 00:18:40.975101 10171 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dec00cfa872446b9bc1cb760c0ef221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260620 00:18:40.975255 10171 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:40.975281 10171 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0dec00cfa872446b9bc1cb760c0ef221, State: Initialized, Role: FOLLOWER
I20260620 00:18:40.975409 10171 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dec00cfa872446b9bc1cb760c0ef221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260620 00:18:40.975773 10171 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:40.975807 10171 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:40.975845 10171 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:40.976508 10171 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dec00cfa872446b9bc1cb760c0ef221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260620 00:18:40.976603 10171 leader_election.cc:302] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0dec00cfa872446b9bc1cb760c0ef221; no voters:
I20260620 00:18:40.976763 10171 leader_election.cc:288] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:40.976796 10176 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:40.976907 10176 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [term 1 LEADER]: Becoming Leader. State: Replica: 0dec00cfa872446b9bc1cb760c0ef221, State: Running, Role: LEADER
I20260620 00:18:40.977005 10176 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dec00cfa872446b9bc1cb760c0ef221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } }
I20260620 00:18:40.977059 10171 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:40.977367 10178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0dec00cfa872446b9bc1cb760c0ef221. Latest consensus state: current_term: 1 leader_uuid: "0dec00cfa872446b9bc1cb760c0ef221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dec00cfa872446b9bc1cb760c0ef221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } } }
I20260620 00:18:40.977473 10178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:40.977627 10177 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0dec00cfa872446b9bc1cb760c0ef221" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0dec00cfa872446b9bc1cb760c0ef221" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34489 } } }
I20260620 00:18:40.977763 10177 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:40.977986 10185 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:18:40.978492 10185 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:40.980129 10185 catalog_manager.cc:1347] Generated new cluster ID: 9e5e9353c28940d69b6e2f9942c731dd
I20260620 00:18:40.980221 10185 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:41.006193 10185 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:41.006433 10185 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:41.015684 10185 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0dec00cfa872446b9bc1cb760c0ef221: Generated new TSK 0
I20260620 00:18:41.015934 10185 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:18:41.024492 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--builtin_ntp_servers=127.0.98.20:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:18:41.115734 10195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:41.116012 10195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:41.116055 10195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:41.117421 10195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:41.117503 10195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:41.117547 10195 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:41.118988 10195 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:41.119282 10195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:41.119614 10195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:41.124845 10203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:41.124832 10201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:41.124832 10200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:41.124862 10195 server_base.cc:1034] running on GCE node
I20260620 00:18:41.125308 10195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:41.125519 10195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:41.126660 10195 hybrid_clock.cc:648] HybridClock initialized: now 1781914721126651 us; error 31 us; skew 500 ppm
I20260620 00:18:41.128057 10195 webserver.cc:460] Webserver started at http://127.0.98.1:43887/ using document root <none> and password file <none>
I20260620 00:18:41.128360 10195 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:41.128412 10195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:41.128587 10195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:41.129703 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "bafcfed92e6d48e785a5dd6642d25ed6"
format_stamp: "Formatted at 2026-06-20 00:18:41 on dist-test-slave-r81b"
server_key: "39e0b6d11e8de605e9e87ffea2901382"
server_key_iv: "abe85856b5905b12e34a43066f7b069c"
server_key_version: "encryptionkey@0"
I20260620 00:18:41.130072 10195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "bafcfed92e6d48e785a5dd6642d25ed6"
format_stamp: "Formatted at 2026-06-20 00:18:41 on dist-test-slave-r81b"
server_key: "39e0b6d11e8de605e9e87ffea2901382"
server_key_iv: "abe85856b5905b12e34a43066f7b069c"
server_key_version: "encryptionkey@0"
I20260620 00:18:41.131413 10195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:41.132239 10209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:41.132483 10195 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:41.132555 10195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "bafcfed92e6d48e785a5dd6642d25ed6"
format_stamp: "Formatted at 2026-06-20 00:18:41 on dist-test-slave-r81b"
server_key: "39e0b6d11e8de605e9e87ffea2901382"
server_key_iv: "abe85856b5905b12e34a43066f7b069c"
server_key_version: "encryptionkey@0"
I20260620 00:18:41.132644 10195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:41.136281 10195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:41.136555 10195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:41.136696 10195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:41.136898 10195 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:41.137200 10195 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:41.137234 10195 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:41.137282 10195 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:41.137308 10195 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:41.143009 10195 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42165
I20260620 00:18:41.143083 10322 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42165 every 8 connection(s)
I20260620 00:18:41.143450 10195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:18:41.146651 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:41.144234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54827 (local address 127.0.98.62:34489)
0620 00:18:41.144313 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:41.144316 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:41.144582 (+ 266us) server_negotiation.cc:407] Connection header received
0620 00:18:41.145048 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:41.145049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:41.145055 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:41.145105 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:41.146372 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:41.146374 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:41.146374 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:41.146378 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:41.146402 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:41.146406 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:41.146470 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:41.146577 (+ 107us) server_negotiation.cc:299] Negotiation successful
0620 00:18:41.146593 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:18:41.146746 10325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:41.144248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34489 (local address 127.0.98.1:54827)
0620 00:18:41.144451 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:41.144484 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:41.144966 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:41.145169 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:41.145172 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:41.145179 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:41.146202 (+ 1023us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:41.146202 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:41.146235 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:41.146266 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:41.146470 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:41.146474 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:41.146477 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:41.146533 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:18:41.146584 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":82,"threads_started":1}
I20260620 00:18:41.147481 10323 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34489
I20260620 00:18:41.147564 10323 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:41.147713 10323 heartbeater.cc:511] Master 127.0.98.62:34489 requested a full tablet report, sending...
I20260620 00:18:41.148440 10135 ts_manager.cc:194] Registered new tserver with Master: bafcfed92e6d48e785a5dd6642d25ed6 (127.0.98.1:42165)
I20260620 00:18:41.150352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 10195
I20260620 00:18:41.150426 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:41.150653 392 external_mini_cluster.cc:1428] Setting key 13ca9cfb34a7cc2fc3c255d488ba39a8
I20260620 00:18:41.151625 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--builtin_ntp_servers=127.0.98.20:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:18:41.232715 10326 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:41.232990 10326 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:41.233031 10326 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:41.234273 10326 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:41.234345 10326 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:41.234390 10326 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:41.235935 10326 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10326
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:41.236248 10326 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:41.236598 10326 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:41.241863 10331 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:41.241868 10334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:41.241879 10332 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:41.241921 10326 server_base.cc:1034] running on GCE node
I20260620 00:18:41.242287 10326 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:41.242545 10326 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:41.243752 10326 hybrid_clock.cc:648] HybridClock initialized: now 1781914721243725 us; error 44 us; skew 500 ppm
I20260620 00:18:41.245235 10326 webserver.cc:460] Webserver started at http://127.0.98.2:46693/ using document root <none> and password file <none>
I20260620 00:18:41.245501 10326 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:41.245553 10326 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:41.245734 10326 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:41.246852 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "95cf469772c8425c8268acc2a9fd9d36"
format_stamp: "Formatted at 2026-06-20 00:18:41 on dist-test-slave-r81b"
server_key: "528a7b0aa8a0098bca1357ab8fb4cf28"
server_key_iv: "c0bd162b864b9ebbafaddda8dff20e57"
server_key_version: "encryptionkey@0"
I20260620 00:18:41.247231 10326 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "95cf469772c8425c8268acc2a9fd9d36"
format_stamp: "Formatted at 2026-06-20 00:18:41 on dist-test-slave-r81b"
server_key: "528a7b0aa8a0098bca1357ab8fb4cf28"
server_key_iv: "c0bd162b864b9ebbafaddda8dff20e57"
server_key_version: "encryptionkey@0"
I20260620 00:18:41.248781 10326 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:41.249635 10340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:41.249854 10326 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:41.249955 10326 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "95cf469772c8425c8268acc2a9fd9d36"
format_stamp: "Formatted at 2026-06-20 00:18:41 on dist-test-slave-r81b"
server_key: "528a7b0aa8a0098bca1357ab8fb4cf28"
server_key_iv: "c0bd162b864b9ebbafaddda8dff20e57"
server_key_version: "encryptionkey@0"
I20260620 00:18:41.250041 10326 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:41.253671 10326 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:41.253954 10326 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:41.254099 10326 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:41.254323 10326 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:41.254627 10326 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:41.254690 10326 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:41.254730 10326 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:41.254758 10326 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:41.260877 10326 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33263
I20260620 00:18:41.260928 10453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33263 every 8 connection(s)
I20260620 00:18:41.261315 10326 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:41.264684 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:41.262091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45583 (local address 127.0.98.62:34489)
0620 00:18:41.262156 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:41.262158 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:41.262436 (+ 278us) server_negotiation.cc:407] Connection header received
0620 00:18:41.263005 (+ 569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:41.263006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:41.263013 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:41.263071 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:41.264422 (+ 1351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:41.264424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:41.264424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:41.264429 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:41.264446 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:41.264450 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:41.264513 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:41.264592 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 00:18:41.264607 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:18:41.264918 10456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:41.262098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34489 (local address 127.0.98.2:45583)
0620 00:18:41.262288 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:41.262325 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:41.262879 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:41.263151 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:41.263155 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:41.263165 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:41.264215 (+ 1050us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:41.264216 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:41.264267 (+ 51us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:41.264307 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:41.264522 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:41.264528 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:41.264531 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:41.264606 (+ 75us) client_negotiation.cc:240] Negotiation successful
0620 00:18:41.264675 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260620 00:18:41.265502 10454 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34489
I20260620 00:18:41.265620 10454 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:41.265753 10454 heartbeater.cc:511] Master 127.0.98.62:34489 requested a full tablet report, sending...
I20260620 00:18:41.266265 10135 ts_manager.cc:194] Registered new tserver with Master: 95cf469772c8425c8268acc2a9fd9d36 (127.0.98.2:33263)
I20260620 00:18:41.266353 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 10326
I20260620 00:18:41.266497 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:41.266709 392 external_mini_cluster.cc:1428] Setting key 78a05120828a23a1e0397d81a59ee502
I20260620 00:18:41.267591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--builtin_ntp_servers=127.0.98.20:42343
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260620 00:18:41.352640 10457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:41.352888 10457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:41.352916 10457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:41.354327 10457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:41.354372 10457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:41.354411 10457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:41.355924 10457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42343
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34489
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:41.356240 10457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:41.356730 10457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:41.363468 10465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:41.363734 10457 server_base.cc:1034] running on GCE node
W20260620 00:18:41.363468 10462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:41.363498 10463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:41.364077 10457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:41.364488 10457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:41.365680 10457 hybrid_clock.cc:648] HybridClock initialized: now 1781914721365667 us; error 47 us; skew 500 ppm
I20260620 00:18:41.367159 10457 webserver.cc:460] Webserver started at http://127.0.98.3:38723/ using document root <none> and password file <none>
I20260620 00:18:41.367465 10457 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:41.367542 10457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:41.367698 10457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:41.368903 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "faef5d040eb94eb99e389b87fa3dfa88"
format_stamp: "Formatted at 2026-06-20 00:18:41 on dist-test-slave-r81b"
server_key: "1f5097bc43b8ab50108b0f0c4fc05ff2"
server_key_iv: "00b68926a33d68511e30cbac179e26c4"
server_key_version: "encryptionkey@0"
I20260620 00:18:41.369284 10457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "faef5d040eb94eb99e389b87fa3dfa88"
format_stamp: "Formatted at 2026-06-20 00:18:41 on dist-test-slave-r81b"
server_key: "1f5097bc43b8ab50108b0f0c4fc05ff2"
server_key_iv: "00b68926a33d68511e30cbac179e26c4"
server_key_version: "encryptionkey@0"
I20260620 00:18:41.370602 10457 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:41.371346 10471 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:41.371568 10457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:41.371680 10457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "faef5d040eb94eb99e389b87fa3dfa88"
format_stamp: "Formatted at 2026-06-20 00:18:41 on dist-test-slave-r81b"
server_key: "1f5097bc43b8ab50108b0f0c4fc05ff2"
server_key_iv: "00b68926a33d68511e30cbac179e26c4"
server_key_version: "encryptionkey@0"
I20260620 00:18:41.371755 10457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:41.375335 10457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:41.375598 10457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:41.375746 10457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:41.375957 10457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:41.376366 10457 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:41.376411 10457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:41.376466 10457 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:41.376493 10457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:41.382552 10457 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38917
I20260620 00:18:41.382580 10584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38917 every 8 connection(s)
I20260620 00:18:41.382948 10457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:18:41.386168 10174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:41.383688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46471 (local address 127.0.98.62:34489)
0620 00:18:41.383755 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:41.383757 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:41.384020 (+ 263us) server_negotiation.cc:407] Connection header received
0620 00:18:41.384587 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:41.384589 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:41.384597 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:41.384676 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:41.385939 (+ 1263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:41.385941 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:41.385941 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:18:41.385945 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:41.385961 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:18:41.385965 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:41.386038 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:41.386100 (+ 62us) server_negotiation.cc:299] Negotiation successful
0620 00:18:41.386113 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:18:41.386353 10587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:41.383687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34489 (local address 127.0.98.3:46471)
0620 00:18:41.383872 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:41.383915 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:41.384471 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:41.384747 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:41.384750 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:41.384759 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:41.385776 (+ 1017us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:18:41.385776 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:41.385809 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:18:41.385839 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:41.386038 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:41.386040 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:41.386043 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:41.386096 (+ 53us) client_negotiation.cc:240] Negotiation successful
0620 00:18:41.386144 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":103,"threads_started":1}
I20260620 00:18:41.386862 10585 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34489
I20260620 00:18:41.386942 10585 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:41.387049 10585 heartbeater.cc:511] Master 127.0.98.62:34489 requested a full tablet report, sending...
I20260620 00:18:41.387365 10135 ts_manager.cc:194] Registered new tserver with Master: faef5d040eb94eb99e389b87fa3dfa88 (127.0.98.3:38917)
I20260620 00:18:41.392817 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 10457
I20260620 00:18:41.392894 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:41.393045 392 external_mini_cluster.cc:1428] Setting key 357abd966992817a3aa1252665ea75d8
I20260620 00:18:41.393956 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260620 00:18:41.395877 10174 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:18:41.395282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42432 (local address 127.0.98.62:34489)
0620 00:18:41.395382 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:41.395384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:41.395498 (+ 114us) server_negotiation.cc:407] Connection header received
0620 00:18:41.395578 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:41.395579 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:41.395584 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:41.395661 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:41.395783 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42432: BlockingRecv error: recv got EOF from 127.0.0.1:42432 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260620 00:18:41.395985 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 10195
I20260620 00:18:41.401015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 10326
I20260620 00:18:41.405678 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 10457
I20260620 00:18:41.410001 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 10104
2026-06-20T00:18:41Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (581 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:41 dist-test-slave-r81b krb5kdc[10595](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:41 dist-test-slave-r81b krb5kdc[10595](info): set up 2 sockets
Jun 20 00:18:41 dist-test-slave-r81b krb5kdc[10595](info): commencing operation
krb5kdc: starting...
W20260620 00:18:42.465113 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:42 dist-test-slave-r81b krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:18:42.591203 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40963
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:42.673597 10608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:42.673882 10608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:42.673925 10608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:42.675140 10608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:18:42.675217 10608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:42.675242 10608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:42.675274 10608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:18:42.675323 10608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:18:42.676988 10608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:42.677287 10608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:42.677524 10608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:42.682710 10616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:42.682705 10614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:42.682812 10608 server_base.cc:1034] running on GCE node
W20260620 00:18:42.682705 10613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:42.683228 10608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:42.683573 10608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:42.684803 10608 hybrid_clock.cc:648] HybridClock initialized: now 1781914722684777 us; error 37 us; skew 500 ppm
Jun 20 00:18:42 dist-test-slave-r81b krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:42.687389 10608 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:18:42.687907 10608 webserver.cc:460] Webserver started at http://127.0.98.62:33721/ using document root <none> and password file <none>
I20260620 00:18:42.688125 10608 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:42.688236 10608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:42.688349 10608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:42.689488 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "3c24a59371134621b4f766d5d4caa9e3"
format_stamp: "Formatted at 2026-06-20 00:18:42 on dist-test-slave-r81b"
server_key: "c4d4b997532c836c7e6a96bc7859f42c"
server_key_iv: "8bfb30e6f2697dd692c67aad527df634"
server_key_version: "encryptionkey@0"
I20260620 00:18:42.689810 10608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "3c24a59371134621b4f766d5d4caa9e3"
format_stamp: "Formatted at 2026-06-20 00:18:42 on dist-test-slave-r81b"
server_key: "c4d4b997532c836c7e6a96bc7859f42c"
server_key_iv: "8bfb30e6f2697dd692c67aad527df634"
server_key_version: "encryptionkey@0"
I20260620 00:18:42.691165 10608 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:18:42.691901 10623 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:42.692114 10608 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:18:42.692270 10608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "3c24a59371134621b4f766d5d4caa9e3"
format_stamp: "Formatted at 2026-06-20 00:18:42 on dist-test-slave-r81b"
server_key: "c4d4b997532c836c7e6a96bc7859f42c"
server_key_iv: "8bfb30e6f2697dd692c67aad527df634"
server_key_version: "encryptionkey@0"
I20260620 00:18:42.692356 10608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:42.714840 10608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:42.717612 10608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:42.717756 10608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:42.722132 10608 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40963
I20260620 00:18:42.722157 10675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40963 every 8 connection(s)
I20260620 00:18:42.722527 10608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:18:42.723513 10676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:42.725980 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3: Bootstrap starting.
I20260620 00:18:42.726512 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 10608
I20260620 00:18:42.726591 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:18:42.726670 10676 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:42.726800 392 external_mini_cluster.cc:1428] Setting key eefe93bd7906a9465440bc965273de06
I20260620 00:18:42.727027 10676 log.cc:824] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:42.727846 10676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3: No bootstrap required, opened a new log
I20260620 00:18:42.730190 10676 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c24a59371134621b4f766d5d4caa9e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40963 } }
I20260620 00:18:42.730356 10676 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:42.730386 10676 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c24a59371134621b4f766d5d4caa9e3, State: Initialized, Role: FOLLOWER
I20260620 00:18:42.730506 10676 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c24a59371134621b4f766d5d4caa9e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40963 } }
I20260620 00:18:42.730605 10676 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:18:42.730650 10676 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:18:42.730700 10676 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:42.731245 10676 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c24a59371134621b4f766d5d4caa9e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40963 } }
I20260620 00:18:42.731377 10676 leader_election.cc:302] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c24a59371134621b4f766d5d4caa9e3; no voters:
I20260620 00:18:42.731536 10676 leader_election.cc:288] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:18:42.731597 10681 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:42.731774 10676 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:18:42.731765 10681 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [term 1 LEADER]: Becoming Leader. State: Replica: 3c24a59371134621b4f766d5d4caa9e3, State: Running, Role: LEADER
I20260620 00:18:42.731887 10681 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c24a59371134621b4f766d5d4caa9e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40963 } }
I20260620 00:18:42.733764 10682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c24a59371134621b4f766d5d4caa9e3. Latest consensus state: current_term: 1 leader_uuid: "3c24a59371134621b4f766d5d4caa9e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c24a59371134621b4f766d5d4caa9e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40963 } } }
I20260620 00:18:42.733958 10683 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c24a59371134621b4f766d5d4caa9e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c24a59371134621b4f766d5d4caa9e3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40963 } } }
I20260620 00:18:42.734330 10682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:18:42.734333 10683 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3 [sys.catalog]: This master's current role is: LEADER
W20260620 00:18:42.734513 10695 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:18:42.734566 10695 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:18:42.734622 10698 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Jun 20 00:18:42 dist-test-slave-r81b krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914722, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:42.734797 10698 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:18:42.736357 10698 catalog_manager.cc:1347] Generated new cluster ID: 5ff71c90d2f94adaa5c5dcbe14259ec9
I20260620 00:18:42.736433 10698 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:18:42.739202 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:42.727357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58870 (local address 127.0.98.62:40963)
0620 00:18:42.727528 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:42.727537 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:42.727570 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:18:42.728194 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:42.728204 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:42.728278 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:42.728445 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:42.729037 (+ 592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:42.730459 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:42.731299 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:42.732174 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:42.732276 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:42.735366 (+ 3090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:42.735370 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:42.735371 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:42.735383 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:42.737148 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:42.737588 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:42.737591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:42.737592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:42.737697 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:42.737917 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:42.737917 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:42.737918 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:42.738224 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:42.738756 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:42.738832 (+ 76us) server_negotiation.cc:299] Negotiation successful
0620 00:18:42.738977 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":69,"threads_started":1}
I20260620 00:18:42.754323 10698 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:18:42.754890 10698 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:18:42.759433 10698 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3: Generated new TSK 0
I20260620 00:18:42.759631 10698 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:18:42.787533 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40963
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:42.870218 10702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:42.870445 10702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:42.870468 10702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:42.871661 10702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:42.871706 10702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:42.871733 10702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:18:42.873194 10702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:42.873456 10702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:42.873708 10702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:42.879048 10710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:42.879042 10708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:42.879151 10702 server_base.cc:1034] running on GCE node
W20260620 00:18:42.879253 10707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:42.879597 10702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:42.879849 10702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:42.881016 10702 hybrid_clock.cc:648] HybridClock initialized: now 1781914722881002 us; error 29 us; skew 500 ppm
Jun 20 00:18:42 dist-test-slave-r81b krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:42.883713 10702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:18:42.884284 10702 webserver.cc:460] Webserver started at http://127.0.98.1:37271/ using document root <none> and password file <none>
I20260620 00:18:42.884559 10702 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:42.884608 10702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:42.884757 10702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:42.885850 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "d56c1455692b49ecb4654ed935da7886"
format_stamp: "Formatted at 2026-06-20 00:18:42 on dist-test-slave-r81b"
server_key: "11cb8c17511b28bd7312cc70d37fe610"
server_key_iv: "1f226bf2ab10b85bd28f1f7eaeb45559"
server_key_version: "encryptionkey@0"
I20260620 00:18:42.886215 10702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d56c1455692b49ecb4654ed935da7886"
format_stamp: "Formatted at 2026-06-20 00:18:42 on dist-test-slave-r81b"
server_key: "11cb8c17511b28bd7312cc70d37fe610"
server_key_iv: "1f226bf2ab10b85bd28f1f7eaeb45559"
server_key_version: "encryptionkey@0"
I20260620 00:18:42.887629 10702 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:42.888482 10717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:42.888671 10702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:42.888780 10702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "d56c1455692b49ecb4654ed935da7886"
format_stamp: "Formatted at 2026-06-20 00:18:42 on dist-test-slave-r81b"
server_key: "11cb8c17511b28bd7312cc70d37fe610"
server_key_iv: "1f226bf2ab10b85bd28f1f7eaeb45559"
server_key_version: "encryptionkey@0"
I20260620 00:18:42.888864 10702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:42.904709 10702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:42.906688 10702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:42.906865 10702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:42.907107 10702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:42.907430 10702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:42.907490 10702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:42.907534 10702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:42.907559 10702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:42.913748 10702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43189
I20260620 00:18:42.913786 10830 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43189 every 8 connection(s)
I20260620 00:18:42.914152 10702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:18:42 dist-test-slave-r81b krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914722, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:42.923030 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 10702
I20260620 00:18:42.923142 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:18:42.923357 392 external_mini_cluster.cc:1428] Setting key 3be1a63d7b3102975938e65af955cc3a
I20260620 00:18:42.924046 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:42.914912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54113 (local address 127.0.98.62:40963)
0620 00:18:42.914984 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:42.914986 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:42.915200 (+ 214us) server_negotiation.cc:407] Connection header received
0620 00:18:42.915686 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:42.915687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:42.915696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:42.915763 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:42.916810 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:42.917446 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:42.918506 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:42.918725 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:42.918809 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:42.921203 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:42.921205 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:42.921205 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:42.921212 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:42.922467 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:42.922846 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:42.922847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:42.922847 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:42.922908 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:42.923295 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:42.923296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:42.923297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:42.923440 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:42.923512 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:42.923862 (+ 350us) server_negotiation.cc:299] Negotiation successful
0620 00:18:42.923958 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:18:42.924253 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:42.914886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40963 (local address 127.0.98.1:54113)
0620 00:18:42.915068 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:42.915112 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:42.915604 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:42.915845 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:42.915849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:42.916515 (+ 666us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:42.916720 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:42.916725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:42.917529 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:42.917530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:42.918404 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:42.918406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:42.918840 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:42.918841 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:42.918967 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:42.919561 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:42.919577 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:42.921111 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:42.922544 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:42.922549 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:42.922551 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:42.922772 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:42.922995 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:42.922996 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:42.922996 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:42.923110 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:42.923511 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:42.923531 (+ 20us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:42.923705 (+ 174us) client_negotiation.cc:769] Sending connection context
0620 00:18:42.923825 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:18:42.923988 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":110,"threads_started":1}
I20260620 00:18:42.924736 10831 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40963
I20260620 00:18:42.924847 10831 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:42.925029 10831 heartbeater.cc:511] Master 127.0.98.62:40963 requested a full tablet report, sending...
I20260620 00:18:42.925539 10640 ts_manager.cc:194] Registered new tserver with Master: d56c1455692b49ecb4654ed935da7886 (127.0.98.1:43189)
I20260620 00:18:42.926177 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54113
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:18:42.949282 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40963
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:43.032884 10836 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:43.033094 10836 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:43.033130 10836 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:43.034323 10836 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:43.034365 10836 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:43.034394 10836 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:18:43.035853 10836 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10836
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:43.036125 10836 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:43.036387 10836 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:43.041851 10842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:43.041857 10841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:43.041853 10844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:43.042227 10836 server_base.cc:1034] running on GCE node
I20260620 00:18:43.042466 10836 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:43.042718 10836 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:43.043880 10836 hybrid_clock.cc:648] HybridClock initialized: now 1781914723043882 us; error 43 us; skew 500 ppm
Jun 20 00:18:43 dist-test-slave-r81b krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:43.046433 10836 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:18:43.046877 10836 webserver.cc:460] Webserver started at http://127.0.98.2:35891/ using document root <none> and password file <none>
I20260620 00:18:43.047127 10836 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:43.047197 10836 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:43.047354 10836 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:43.048504 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "628c5b34c87d46cbb479376894910a05"
format_stamp: "Formatted at 2026-06-20 00:18:43 on dist-test-slave-r81b"
server_key: "217c1b7e6096a1a181c572720529fc97"
server_key_iv: "e7897cdd6409c7182b7c9851eaa20215"
server_key_version: "encryptionkey@0"
I20260620 00:18:43.048870 10836 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "628c5b34c87d46cbb479376894910a05"
format_stamp: "Formatted at 2026-06-20 00:18:43 on dist-test-slave-r81b"
server_key: "217c1b7e6096a1a181c572720529fc97"
server_key_iv: "e7897cdd6409c7182b7c9851eaa20215"
server_key_version: "encryptionkey@0"
I20260620 00:18:43.050184 10836 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:18:43.050966 10851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:43.051139 10836 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:43.051241 10836 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "628c5b34c87d46cbb479376894910a05"
format_stamp: "Formatted at 2026-06-20 00:18:43 on dist-test-slave-r81b"
server_key: "217c1b7e6096a1a181c572720529fc97"
server_key_iv: "e7897cdd6409c7182b7c9851eaa20215"
server_key_version: "encryptionkey@0"
I20260620 00:18:43.051321 10836 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:43.069729 10836 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:43.072124 10836 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:43.072396 10836 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:43.072639 10836 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:43.072966 10836 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:43.073020 10836 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:43.073062 10836 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:43.073089 10836 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:43.078943 10836 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42005
I20260620 00:18:43.078986 10964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42005 every 8 connection(s)
I20260620 00:18:43.079360 10836 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:18:43.084878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 10836
I20260620 00:18:43.085011 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:18:43.085216 392 external_mini_cluster.cc:1428] Setting key 0b5631544abc8b8babef58582f03d6bd
Jun 20 00:18:43 dist-test-slave-r81b krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:43.089596 10967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.080111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40963 (local address 127.0.98.2:53629)
0620 00:18:43.080316 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:43.080359 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:43.080848 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:43.081093 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:43.081097 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:43.081694 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:43.081902 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.081907 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.082654 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.082655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.083491 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.083493 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.083932 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.083933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.084044 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.084630 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:43.084644 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:43.086156 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:43.088127 (+ 1971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:43.088133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:43.088240 (+ 107us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:43.088483 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:43.088688 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:43.088689 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:43.088689 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:43.088786 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:43.089064 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:43.089068 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:43.089221 (+ 153us) client_negotiation.cc:769] Sending connection context
0620 00:18:43.089301 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:18:43.089425 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":140,"threads_started":1}
I20260620 00:18:43.089604 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.080172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53629 (local address 127.0.98.62:40963)
0620 00:18:43.080229 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:43.080231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:43.080448 (+ 217us) server_negotiation.cc:407] Connection header received
0620 00:18:43.080933 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:43.080934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:43.080944 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:43.081007 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:43.082001 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.082530 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.083619 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.083820 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.083888 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.086252 (+ 2364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:43.086254 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:43.086254 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:43.086262 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:43.088036 (+ 1774us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:43.088554 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:43.088555 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:43.088556 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:43.088618 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:43.088857 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:43.088858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:43.088859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:43.089006 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:43.089069 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:43.089365 (+ 296us) server_negotiation.cc:299] Negotiation successful
0620 00:18:43.089461 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:18:43.090207 10965 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40963
I20260620 00:18:43.090354 10965 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:43.090551 10965 heartbeater.cc:511] Master 127.0.98.62:40963 requested a full tablet report, sending...
I20260620 00:18:43.090994 10640 ts_manager.cc:194] Registered new tserver with Master: 628c5b34c87d46cbb479376894910a05 (127.0.98.2:42005)
I20260620 00:18:43.091544 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53629
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:18:43.112917 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40963
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:18:43.194284 10970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:18:43.194492 10970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:18:43.194517 10970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:18:43.195783 10970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:18:43.195827 10970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:18:43.195854 10970 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:18:43.197340 10970 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:18:43.197602 10970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:18:43.197849 10970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:18:43.203006 10978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:18:43.202996 10975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:43.203150 10970 server_base.cc:1034] running on GCE node
W20260620 00:18:43.202996 10976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:18:43.203476 10970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:18:43.203749 10970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:18:43.204923 10970 hybrid_clock.cc:648] HybridClock initialized: now 1781914723204890 us; error 51 us; skew 500 ppm
Jun 20 00:18:43 dist-test-slave-r81b krb5kdc[10595](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:18:43.207661 10970 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:18:43.208214 10970 webserver.cc:460] Webserver started at http://127.0.98.3:46293/ using document root <none> and password file <none>
I20260620 00:18:43.208514 10970 fs_manager.cc:359] Metadata directory not provided
I20260620 00:18:43.208562 10970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:18:43.208714 10970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:18:43.209824 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "8605f72538db40798043400ce6ac46d8"
format_stamp: "Formatted at 2026-06-20 00:18:43 on dist-test-slave-r81b"
server_key: "c01c796e49e361aa0cbf66643a08f5c3"
server_key_iv: "f2e9caf36f8f25c7f83c38378b82a812"
server_key_version: "encryptionkey@0"
I20260620 00:18:43.210203 10970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8605f72538db40798043400ce6ac46d8"
format_stamp: "Formatted at 2026-06-20 00:18:43 on dist-test-slave-r81b"
server_key: "c01c796e49e361aa0cbf66643a08f5c3"
server_key_iv: "f2e9caf36f8f25c7f83c38378b82a812"
server_key_version: "encryptionkey@0"
I20260620 00:18:43.211612 10970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:18:43.212507 10985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:43.212791 10970 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:18:43.212905 10970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "8605f72538db40798043400ce6ac46d8"
format_stamp: "Formatted at 2026-06-20 00:18:43 on dist-test-slave-r81b"
server_key: "c01c796e49e361aa0cbf66643a08f5c3"
server_key_iv: "f2e9caf36f8f25c7f83c38378b82a812"
server_key_version: "encryptionkey@0"
I20260620 00:18:43.212981 10970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:18:43.240690 10970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:18:43.242710 10970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:18:43.242887 10970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:18:43.243121 10970 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:18:43.243441 10970 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:18:43.243498 10970 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:43.243541 10970 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:18:43.243570 10970 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:18:43.249912 10970 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34593
I20260620 00:18:43.250000 11098 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34593 every 8 connection(s)
I20260620 00:18:43.250331 10970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:18:43 dist-test-slave-r81b krb5kdc[10595](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914723, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:18:43.258087 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 10970
I20260620 00:18:43.258193 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:18:43.258375 392 external_mini_cluster.cc:1428] Setting key ea36534463c94b8026954c4e1022dfe9
I20260620 00:18:43.260347 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.251178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46189 (local address 127.0.98.62:40963)
0620 00:18:43.251269 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:43.251271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:43.251435 (+ 164us) server_negotiation.cc:407] Connection header received
0620 00:18:43.251894 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:43.251895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:43.251906 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:43.251971 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:43.252963 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.253471 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.254589 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.254786 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.254843 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.257136 (+ 2293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:43.257137 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:43.257138 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:43.257145 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:43.258744 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:43.259302 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:43.259303 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:43.259304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:43.259385 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:43.259672 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:43.259672 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:43.259673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:43.259798 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:43.259848 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:43.260210 (+ 362us) server_negotiation.cc:299] Negotiation successful
0620 00:18:43.260284 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260620 00:18:43.260612 11101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.251117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40963 (local address 127.0.98.3:46189)
0620 00:18:43.251277 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:43.251320 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:43.251804 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:43.252025 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:43.252028 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:43.252682 (+ 654us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:18:43.252878 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.252883 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.253576 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.253577 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.254511 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.254514 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.254864 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.254864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.254965 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.255633 (+ 668us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:18:43.255642 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:18:43.257048 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:18:43.258863 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:43.258870 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:43.258873 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:43.259228 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:43.259458 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:18:43.259459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:18:43.259459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:18:43.259609 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:18:43.259847 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:18:43.259850 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:18:43.260045 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 00:18:43.260175 (+ 130us) client_negotiation.cc:240] Negotiation successful
0620 00:18:43.260337 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":90,"threads_started":1}
I20260620 00:18:43.261103 11099 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40963
I20260620 00:18:43.261204 11099 heartbeater.cc:464] Registering TS with master...
I20260620 00:18:43.261389 11099 heartbeater.cc:511] Master 127.0.98.62:40963 requested a full tablet report, sending...
I20260620 00:18:43.261776 10640 ts_manager.cc:194] Registered new tserver with Master: 8605f72538db40798043400ce6ac46d8 (127.0.98.3:34593)
I20260620 00:18:43.262159 10640 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46189
I20260620 00:18:43.270761 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:18:43.278476 10680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.272289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58884 (local address 127.0.98.62:40963)
0620 00:18:43.272361 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:43.272363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:43.272447 (+ 84us) server_negotiation.cc:407] Connection header received
0620 00:18:43.272526 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:43.272527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:43.272538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:43.272610 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:18:43.273223 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.273726 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.274472 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.274664 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.274762 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.275989 (+ 1227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:18:43.275991 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:18:43.275991 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:18:43.275999 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:18:43.277239 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:43.277604 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:43.277605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:43.277606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:43.277682 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:18:43.277890 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:18:43.277891 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:18:43.277892 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:18:43.278058 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:18:43.278172 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:43.278278 (+ 106us) server_negotiation.cc:299] Negotiation successful
0620 00:18:43.278368 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:18:43.280458 10640 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58884:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:18:43.282120 10640 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:18:43.289036 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.284975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50378 (local address 127.0.98.3:34593)
0620 00:18:43.285387 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:43.285392 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:43.285400 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:18:43.285455 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:43.285457 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:43.285475 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:43.285780 (+ 305us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:43.285856 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.286614 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.288023 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.288627 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.288720 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.288786 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:43.288903 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 00:18:43.288942 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":377,"thread_start_us":95,"threads_started":1}
I20260620 00:18:43.289021 11110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.284894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34593 (local address 127.0.0.1:50378)
0620 00:18:43.285151 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:43.285161 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:43.285203 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:43.285586 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:43.285589 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:43.285592 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:43.285672 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.285675 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.286719 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.286721 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.287928 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.287931 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.288718 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.288719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.288797 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.288811 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:18:43.288887 (+ 76us) client_negotiation.cc:240] Negotiation successful
0620 00:18:43.288942 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260620 00:18:43.289390 11113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.285102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43189 (local address 127.0.0.1:53404)
0620 00:18:43.285517 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:43.285529 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:43.285585 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:43.285760 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:43.285761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:43.285763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:43.285831 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.285833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.287209 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.287210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.288089 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.288091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.289159 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.289160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.289211 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.289215 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:43.289272 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:18:43.289312 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":70,"threads_started":1}
I20260620 00:18:43.289602 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.284894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42005 (local address 127.0.0.1:42666)
0620 00:18:43.285056 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:43.285100 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:43.285181 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:43.285872 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:43.285873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:43.285875 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:43.285914 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.285916 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.287149 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.287150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.288201 (+ 1051us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.288203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.289090 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.289091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.289364 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.289367 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:18:43.289399 (+ 32us) client_negotiation.cc:240] Negotiation successful
0620 00:18:43.289441 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":69,"threads_started":1}
I20260620 00:18:43.289705 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.285311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53404 (local address 127.0.98.1:43189)
0620 00:18:43.285515 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:43.285519 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:43.285574 (+ 55us) server_negotiation.cc:407] Connection header received
0620 00:18:43.285649 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:43.285651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:43.285665 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:43.285747 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:43.285916 (+ 169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.287089 (+ 1173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.288253 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.288978 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.289053 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.289130 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:43.289567 (+ 437us) server_negotiation.cc:299] Negotiation successful
0620 00:18:43.289612 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
I20260620 00:18:43.289942 11033 tablet_service.cc:1467] Processing CreateTablet for tablet 03a69e18cd44418ca44488cccc178f9a (DEFAULT_TABLE table=test-table [id=bd86eec7691f4f339b1dde7119f3e26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:43.290354 11033 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a69e18cd44418ca44488cccc178f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:43.290383 10765 tablet_service.cc:1467] Processing CreateTablet for tablet 03a69e18cd44418ca44488cccc178f9a (DEFAULT_TABLE table=test-table [id=bd86eec7691f4f339b1dde7119f3e26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:43.290686 10765 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a69e18cd44418ca44488cccc178f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:43.291317 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.285157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42666 (local address 127.0.98.2:42005)
0620 00:18:43.285703 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:43.285706 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:43.285736 (+ 30us) server_negotiation.cc:407] Connection header received
0620 00:18:43.285774 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:43.285776 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:43.285791 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:43.285877 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:43.285960 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.287038 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.288285 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.289002 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.291030 (+ 2028us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.291134 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:43.291201 (+ 67us) server_negotiation.cc:299] Negotiation successful
0620 00:18:43.291248 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":69,"threads_started":1}
I20260620 00:18:43.291841 10899 tablet_service.cc:1467] Processing CreateTablet for tablet 03a69e18cd44418ca44488cccc178f9a (DEFAULT_TABLE table=test-table [id=bd86eec7691f4f339b1dde7119f3e26f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:18:43.292177 10899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 03a69e18cd44418ca44488cccc178f9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:18:43.293926 11116 tablet_bootstrap.cc:492] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886: Bootstrap starting.
I20260620 00:18:43.294700 11116 tablet_bootstrap.cc:654] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:43.295161 11118 tablet_bootstrap.cc:492] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: Bootstrap starting.
I20260620 00:18:43.295344 11116 log.cc:824] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:43.295652 11117 tablet_bootstrap.cc:492] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: Bootstrap starting.
I20260620 00:18:43.295958 11118 tablet_bootstrap.cc:654] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:43.296185 11116 tablet_bootstrap.cc:492] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886: No bootstrap required, opened a new log
I20260620 00:18:43.296339 11116 ts_tablet_manager.cc:1397] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:18:43.296446 11117 tablet_bootstrap.cc:654] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: Neither blocks nor log segments found. Creating new log.
I20260620 00:18:43.296718 11118 log.cc:824] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:43.296995 11117 log.cc:824] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:18:43.297533 11118 tablet_bootstrap.cc:492] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: No bootstrap required, opened a new log
I20260620 00:18:43.297610 11118 ts_tablet_manager.cc:1397] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 00:18:43.298589 11116 raft_consensus.cc:348] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } }
I20260620 00:18:43.298784 11116 raft_consensus.cc:374] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:43.298835 11116 raft_consensus.cc:729] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d56c1455692b49ecb4654ed935da7886, State: Initialized, Role: FOLLOWER
I20260620 00:18:43.298956 11116 consensus_queue.cc:260] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } }
I20260620 00:18:43.299289 11116 ts_tablet_manager.cc:1428] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:18:43.299413 10831 heartbeater.cc:503] Master 127.0.98.62:40963 was elected leader, sending a full tablet report...
I20260620 00:18:43.299597 11117 tablet_bootstrap.cc:492] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: No bootstrap required, opened a new log
I20260620 00:18:43.299674 11117 ts_tablet_manager.cc:1397] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260620 00:18:43.299686 11118 raft_consensus.cc:348] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } }
I20260620 00:18:43.299907 11118 raft_consensus.cc:374] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:43.299963 11118 raft_consensus.cc:729] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 628c5b34c87d46cbb479376894910a05, State: Initialized, Role: FOLLOWER
I20260620 00:18:43.300069 11118 consensus_queue.cc:260] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } }
I20260620 00:18:43.300379 11118 ts_tablet_manager.cc:1428] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:18:43.300513 10965 heartbeater.cc:503] Master 127.0.98.62:40963 was elected leader, sending a full tablet report...
I20260620 00:18:43.301767 11117 raft_consensus.cc:348] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } }
I20260620 00:18:43.301906 11117 raft_consensus.cc:374] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:18:43.301927 11117 raft_consensus.cc:729] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8605f72538db40798043400ce6ac46d8, State: Initialized, Role: FOLLOWER
I20260620 00:18:43.302060 11117 consensus_queue.cc:260] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } }
I20260620 00:18:43.302364 11099 heartbeater.cc:503] Master 127.0.98.62:40963 was elected leader, sending a full tablet report...
I20260620 00:18:43.302371 11117 ts_tablet_manager.cc:1428] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260620 00:18:43.329993 10966 tablet.cc:2369] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:18:43.408625 11122 raft_consensus.cc:482] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:18:43.408756 11122 raft_consensus.cc:504] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } }
I20260620 00:18:43.409142 11122 leader_election.cc:288] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8605f72538db40798043400ce6ac46d8 (127.0.98.3:34593), 628c5b34c87d46cbb479376894910a05 (127.0.98.2:42005)
I20260620 00:18:43.413364 11112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.409391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52645 (local address 127.0.98.3:34593)
0620 00:18:43.409452 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:43.409455 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:43.409464 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:18:43.409496 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:43.409497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:43.409530 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:43.409609 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:43.409806 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.410569 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.412314 (+ 1745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.412914 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.412998 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.413010 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:43.413248 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 00:18:43.413291 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:18:43.413364 11114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.409484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50695 (local address 127.0.98.2:42005)
0620 00:18:43.409552 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:43.409554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:43.409615 (+ 61us) server_negotiation.cc:407] Connection header received
0620 00:18:43.409677 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:43.409678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:43.409703 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:43.409747 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:18:43.409903 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.410590 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.412314 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.412923 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.412998 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.413010 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:43.413248 (+ 238us) server_negotiation.cc:299] Negotiation successful
0620 00:18:43.413291 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:18:43.413380 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.409309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34593 (local address 127.0.98.1:52645)
0620 00:18:43.409365 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:43.409378 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:43.409423 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:43.409621 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:43.409621 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:43.409623 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:43.409706 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.409708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.410690 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.410691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.412204 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.412208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.413031 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.413032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.413134 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.413139 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:18:43.413207 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 00:18:43.413263 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36,"mutex_wait_us":144}
I20260620 00:18:43.413385 11125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.409353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42005 (local address 127.0.98.1:50695)
0620 00:18:43.409517 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:43.409528 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:18:43.409618 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:18:43.409776 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:18:43.409777 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:18:43.409779 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:18:43.409842 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.409842 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.410690 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.410691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.412204 (+ 1513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:18:43.412208 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.413031 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.413032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:18:43.413169 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.413173 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:18:43.413227 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:18:43.413262 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":106,"thread_start_us":79,"threads_started":1}
I20260620 00:18:43.413836 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a69e18cd44418ca44488cccc178f9a" candidate_uuid: "d56c1455692b49ecb4654ed935da7886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8605f72538db40798043400ce6ac46d8" is_pre_election: true
I20260620 00:18:43.414036 11053 raft_consensus.cc:2454] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56c1455692b49ecb4654ed935da7886 in term 0.
I20260620 00:18:43.414032 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a69e18cd44418ca44488cccc178f9a" candidate_uuid: "d56c1455692b49ecb4654ed935da7886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628c5b34c87d46cbb479376894910a05" is_pre_election: true
I20260620 00:18:43.414243 10919 raft_consensus.cc:2454] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d56c1455692b49ecb4654ed935da7886 in term 0.
I20260620 00:18:43.414317 10719 leader_election.cc:302] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8605f72538db40798043400ce6ac46d8, d56c1455692b49ecb4654ed935da7886; no voters:
I20260620 00:18:43.414465 11122 raft_consensus.cc:2781] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:18:43.414542 11122 raft_consensus.cc:482] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:18:43.414580 11122 raft_consensus.cc:3037] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 0 FOLLOWER]: Advancing to term 1
W20260620 00:18:43.414873 10832 tablet.cc:2369] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:18:43.415545 11122 raft_consensus.cc:504] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } }
I20260620 00:18:43.415798 11122 leader_election.cc:288] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [CANDIDATE]: Term 1 election: Requested vote from peers 8605f72538db40798043400ce6ac46d8 (127.0.98.3:34593), 628c5b34c87d46cbb479376894910a05 (127.0.98.2:42005)
I20260620 00:18:43.415892 11053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a69e18cd44418ca44488cccc178f9a" candidate_uuid: "d56c1455692b49ecb4654ed935da7886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8605f72538db40798043400ce6ac46d8"
I20260620 00:18:43.415910 10919 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "03a69e18cd44418ca44488cccc178f9a" candidate_uuid: "d56c1455692b49ecb4654ed935da7886" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "628c5b34c87d46cbb479376894910a05"
I20260620 00:18:43.415972 11053 raft_consensus.cc:3037] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:43.416018 10919 raft_consensus.cc:3037] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:18:43.416754 11053 raft_consensus.cc:2454] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56c1455692b49ecb4654ed935da7886 in term 1.
I20260620 00:18:43.416915 10719 leader_election.cc:302] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8605f72538db40798043400ce6ac46d8, d56c1455692b49ecb4654ed935da7886; no voters:
I20260620 00:18:43.416978 10919 raft_consensus.cc:2454] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d56c1455692b49ecb4654ed935da7886 in term 1.
I20260620 00:18:43.417019 11122 raft_consensus.cc:2781] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:18:43.417197 11122 raft_consensus.cc:686] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [term 1 LEADER]: Becoming Leader. State: Replica: d56c1455692b49ecb4654ed935da7886, State: Running, Role: LEADER
I20260620 00:18:43.417337 11122 consensus_queue.cc:237] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } }
I20260620 00:18:43.418198 10640 catalog_manager.cc:5495] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 reported cstate change: term changed from 0 to 1, leader changed from <none> to d56c1455692b49ecb4654ed935da7886 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "d56c1455692b49ecb4654ed935da7886" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d56c1455692b49ecb4654ed935da7886" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43189 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 } health_report { overall_health: UNKNOWN } } }
I20260620 00:18:43.437352 11115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.434404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53416 (local address 127.0.98.1:43189)
0620 00:18:43.434465 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:18:43.434468 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:18:43.434479 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:18:43.434564 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:18:43.434565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:18:43.434589 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:18:43.434693 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:18:43.434886 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.435459 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.436424 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:18:43.436661 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:18:43.436750 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:18:43.436935 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:18:43.437065 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:18:43.437186 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0620 00:18:43.437206 (+ 20us) server_negotiation.cc:299] Negotiation successful
0620 00:18:43.437228 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:18:43.439286 11053 raft_consensus.cc:1261] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [term 1 FOLLOWER]: Refusing update from remote peer d56c1455692b49ecb4654ed935da7886: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:43.439286 10919 raft_consensus.cc:1261] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [term 1 FOLLOWER]: Refusing update from remote peer d56c1455692b49ecb4654ed935da7886: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:18:43.439633 11122 consensus_queue.cc:1035] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8605f72538db40798043400ce6ac46d8" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:43.439812 11126 consensus_queue.cc:1035] T 03a69e18cd44418ca44488cccc178f9a P d56c1455692b49ecb4654ed935da7886 [LEADER]: Connected to new peer: Peer: permanent_uuid: "628c5b34c87d46cbb479376894910a05" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:18:43.442627 11130 mvcc.cc:205] Tried to move back new op lower bound from 7298722707204513792 to 7298722707118002176. Current Snapshot: MvccSnapshot[applied={T|T < 7298722707204513792}]
I20260620 00:18:43.442958 11129 mvcc.cc:205] Tried to move back new op lower bound from 7298722707204513792 to 7298722707118002176. Current Snapshot: MvccSnapshot[applied={T|T < 7298722707204513792 or (T in {7298722707204513792})}]
I20260620 00:18:43.444568 10640 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:43.444662 10640 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58884:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:18:43.445372 10640 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3c24a59371134621b4f766d5d4caa9e3: Sending DeleteTablet for 3 replicas of tablet 03a69e18cd44418ca44488cccc178f9a
I20260620 00:18:43.445762 11033 tablet_service.cc:1514] Processing DeleteTablet for tablet 03a69e18cd44418ca44488cccc178f9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50378
I20260620 00:18:43.445771 10899 tablet_service.cc:1514] Processing DeleteTablet for tablet 03a69e18cd44418ca44488cccc178f9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42666
I20260620 00:18:43.445797 10765 tablet_service.cc:1514] Processing DeleteTablet for tablet 03a69e18cd44418ca44488cccc178f9a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:18:43 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53404
I20260620 00:18:43.446030 11139 tablet_replica.cc:330] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: stopping tablet replica
I20260620 00:18:43.446053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 10702
I20260620 00:18:43.446148 11141 tablet_replica.cc:330] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: stopping tablet replica
I20260620 00:18:43.446238 11141 raft_consensus.cc:2229] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:43.446347 11141 raft_consensus.cc:2258] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:43.446491 11139 raft_consensus.cc:2229] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:18:43.446719 11139 raft_consensus.cc:2258] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:18:43.447139 11141 ts_tablet_manager.cc:1905] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:43.447149 11139 ts_tablet_manager.cc:1905] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:18:43.448576 11141 ts_tablet_manager.cc:1918] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:43.448624 11141 log.cc:1196] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/03a69e18cd44418ca44488cccc178f9a
I20260620 00:18:43.448836 11141 ts_tablet_manager.cc:1939] T 03a69e18cd44418ca44488cccc178f9a P 628c5b34c87d46cbb479376894910a05: Deleting consensus metadata
I20260620 00:18:43.449160 11139 ts_tablet_manager.cc:1918] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:18:43.449178 10627 catalog_manager.cc:4841] TS 628c5b34c87d46cbb479376894910a05 (127.0.98.2:42005): tablet 03a69e18cd44418ca44488cccc178f9a (table test-table [id=bd86eec7691f4f339b1dde7119f3e26f]) successfully deleted
I20260620 00:18:43.449227 11139 log.cc:1196] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/03a69e18cd44418ca44488cccc178f9a
I20260620 00:18:43.449499 11139 ts_tablet_manager.cc:1939] T 03a69e18cd44418ca44488cccc178f9a P 8605f72538db40798043400ce6ac46d8: Deleting consensus metadata
I20260620 00:18:43.449932 10625 catalog_manager.cc:4841] TS 8605f72538db40798043400ce6ac46d8 (127.0.98.3:34593): tablet 03a69e18cd44418ca44488cccc178f9a (table test-table [id=bd86eec7691f4f339b1dde7119f3e26f]) successfully deleted
W20260620 00:18:43.453243 10625 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43189 (error 108)
I20260620 00:18:43.453438 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 10836
I20260620 00:18:43.454293 11111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:18:43.454154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43189 (local address 127.0.0.1:53430)
0620 00:18:43.454217 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0620 00:18:43.454262 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 00:18:43.454505 10625 catalog_manager.cc:4584] TS d56c1455692b49ecb4654ed935da7886 (127.0.98.1:43189): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 03a69e18cd44418ca44488cccc178f9a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43189: connect: Connection refused (error 111)
I20260620 00:18:43.458645 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 10970
I20260620 00:18:43.463625 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 10608
2026-06-20T00:18:43Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2054 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:18:43 dist-test-slave-r81b krb5kdc[11145](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:18:43 dist-test-slave-r81b krb5kdc[11145](info): set up 2 sockets
Jun 20 00:18:43 dist-test-slave-r81b krb5kdc[11145](info): commencing operation
krb5kdc: starting...
W20260620 00:18:44.517176 392 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:18:44 dist-test-slave-r81b krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914724, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:18:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:18:44Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 00:18:44.688396 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbwaPxd/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/postgres -l logfile start
2026-06-20 00:18:47.350 UTC [11175] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 00:18:47.350 UTC [11175] LOG: listening on IPv4 address "127.0.98.20", port 41697
2026-06-20 00:18:47.355 UTC [11175] LOG: listening on Unix socket "/tmp/.s.PGSQL.41697"
2026-06-20 00:18:47.378 UTC [11178] LOG: database system was shut down at 2026-06-20 00:18:45 UTC
2026-06-20 00:18:47.382 UTC [11175] LOG: database system is ready to accept connections
I20260620 00:18:48.325295 392 mini_postgres.cc:96] Postgres bound to 41697
2026-06-20 00:18:48.331 UTC [11186] FATAL: database "slave" does not exist
127.0.98.20:41697 - accepting connections
I20260620 00:18:48.332177 392 mini_ranger.cc:161] Starting Ranger...
I20260620 00:18:48.347024 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 00:18:48.888792 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 00:18:48.888901 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-admin
2026-06-20 00:18:49,175 [I] DB FLAVOR :POSTGRES
2026-06-20 00:18:49,175 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:18:49,175 [I] Checking connection..
2026-06-20 00:18:49,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 00:18:49,441 [I] Checking connection passed.
2026-06-20 00:18:49,441 [I] --------- Verifying version history table ---------
2026-06-20 00:18:49,441 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:18:49,694 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:18:49,694 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:18:49,936 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:18:49,936 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 00:18:49,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 00:18:50.203 UTC [11250] WARNING: there is no transaction in progress
2026-06-20 00:18:50,210 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 00:18:50,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:18:50,468 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 00:18:50,468 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:18:50,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 00:18:50,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 00:18:50,955 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r81b.c.gcp-upstream.internal','N') ;"
2026-06-20 00:18:51,206 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:18:51,448 [I] Table x_portal_user does not exist in database ranger
2026-06-20 00:18:51,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:18:51,695 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 00:18:51,695 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:18:51,949 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:18:52,207 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 00:18:52,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 00:18:53.236 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.254 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.272 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.290 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.308 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.326 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.386 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.394 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.411 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.422 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.433 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.447 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.457 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.464 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.471 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:53.479 UTC [11358] WARNING: there is no transaction in progress
2026-06-20 00:18:54,216 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 00:18:54,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 00:18:54,475 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 00:18:54,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:18:54,727 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 00:18:54,727 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:18:54,982 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 00:18:54,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:18:55,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:18:55,486 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r81b.c.gcp-upstream.internal';"
2026-06-20 00:18:55,738 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 00:18:55,739 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 00:18:56,000 [I] DB_PATCHES have already been applied
I20260620 00:18:56.006495 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbwaPxd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 00:18:56.006572 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 00:18:56.009867 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:34455
Jun 20, 2026 12:18:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:18:56 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:18:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:18:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:18:56 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:18:57 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:18:57 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 00:18:57.150255 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.818s user 0.009s sys 0.000s
Jun 20, 2026 12:19:17 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:19:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 12:19:17 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:19:17 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:19:18 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 00:19:19.328290 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 00:19:19 dist-test-slave-r81b krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914759, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 00:19:19.436594 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 00:19:19.448652 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 00:19:20.451422 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 00:19:20.451540 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-kms
I20260620 00:19:20.451567 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41697
2026-06-20 00:19:20,635 [I] DB FLAVOR :POSTGRES
2026-06-20 00:19:20,635 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:19:20,635 [I] Checking connection
2026-06-20 00:19:20,635 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 00:19:20,893 [I] connection success
2026-06-20 00:19:20,893 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 00:19:20,893 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 00:19:20,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 00:19:21,166 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 00:19:21,166 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:19:21,167 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 00:19:21,167 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41697/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 00:19:21,431 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 00:19:21.939229 392 mini_ranger_kms.cc:310] Created kms service
I20260620 00:19:22.138094 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 00:19:22.184973 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 00:19:22 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914759, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:19:22.586768 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 00:19:22.586892 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 00:19:22.586912 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 00:19:22.590252 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11538
I20260620 00:19:22.590384 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:52187
Jun 20, 2026 12:19:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:19:23 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:19:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:19:23 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:19:24 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:19:24 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 00:19:24.844326 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.408s user 0.000s sys 0.167s
I20260620 00:19:24.844440 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 00:19:24.844537 392 mini_ranger_kms.cc:401] 127.0.98.20:52187/kms/v1/keys
Jun 20 00:19:24 dist-test-slave-r81b krb5kdc[11145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914764, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 00:19:25 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914764, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:19:25 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914764, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:19:28 dist-test-slave-r81b krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:19:28.174551 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46509
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46509
--ranger_config_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52187/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:19:28.264699 11589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:19:28.264962 11589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:19:28.265000 11589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:19:28.266248 11589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:19:28.266319 11589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:19:28.266345 11589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:19:28.266376 11589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:19:28.266405 11589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:19:28.268249 11589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:52187/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46509
--ranger_config_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:19:28.268524 11589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:19:28.268745 11589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:19:28.274317 11597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:19:28.274334 11594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:19:28.274432 11595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:19:28.274485 11589 server_base.cc:1034] running on GCE node
I20260620 00:19:28.274761 11589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:19:28.275108 11589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:19:28.276269 11589 hybrid_clock.cc:648] HybridClock initialized: now 1781914768276250 us; error 36 us; skew 500 ppm
Jun 20 00:19:28 dist-test-slave-r81b krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:19:28.279261 11589 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:19:28.279753 11589 webserver.cc:460] Webserver started at http://127.0.98.62:40523/ using document root <none> and password file <none>
I20260620 00:19:28.279974 11589 fs_manager.cc:359] Metadata directory not provided
I20260620 00:19:28.280014 11589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:19:28.280140 11589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:19:28 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914768, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:19:28.339011 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "ab8dfaac12fa413ea65468609ab2f605"
format_stamp: "Formatted at 2026-06-20 00:19:28 on dist-test-slave-r81b"
server_key: "1673709b5fa4422c8f21d56254c152ac"
server_key_iv: "a6e1c2ae79055233ed6ba8588919d074"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:28.339484 11589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "ab8dfaac12fa413ea65468609ab2f605"
format_stamp: "Formatted at 2026-06-20 00:19:28 on dist-test-slave-r81b"
server_key: "1673709b5fa4422c8f21d56254c152ac"
server_key_iv: "a6e1c2ae79055233ed6ba8588919d074"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:28.341398 11589 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 00:19:28.353600 11604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:28.354238 11589 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:19:28.354368 11589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "ab8dfaac12fa413ea65468609ab2f605"
format_stamp: "Formatted at 2026-06-20 00:19:28 on dist-test-slave-r81b"
server_key: "1673709b5fa4422c8f21d56254c152ac"
server_key_iv: "a6e1c2ae79055233ed6ba8588919d074"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:28.354470 11589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:19:28.367012 11589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:19:28.369204 11589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:19:28.369370 11589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:19:28.374771 11589 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46509
I20260620 00:19:28.374812 11656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46509 every 8 connection(s)
I20260620 00:19:28.375264 11589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:19:28.375970 11657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:19:28.378304 11657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605: Bootstrap starting.
I20260620 00:19:28.378901 11657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605: Neither blocks nor log segments found. Creating new log.
I20260620 00:19:28.379168 11657 log.cc:824] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605: Log is configured to *not* fsync() on all Append() calls
I20260620 00:19:28.379868 11657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605: No bootstrap required, opened a new log
I20260620 00:19:28.381305 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 11589
I20260620 00:19:28.381424 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:19:28.381641 392 external_mini_cluster.cc:1428] Setting key 3c595ab1758e6806a50bff487eeb7886
I20260620 00:19:28.382301 11657 raft_consensus.cc:348] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8dfaac12fa413ea65468609ab2f605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260620 00:19:28.382495 11657 raft_consensus.cc:374] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:19:28.382531 11657 raft_consensus.cc:729] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8dfaac12fa413ea65468609ab2f605, State: Initialized, Role: FOLLOWER
I20260620 00:19:28.382702 11657 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8dfaac12fa413ea65468609ab2f605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260620 00:19:28.382799 11657 raft_consensus.cc:388] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:19:28.382853 11657 raft_consensus.cc:482] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:19:28.382910 11657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:19:28.383685 11657 raft_consensus.cc:504] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8dfaac12fa413ea65468609ab2f605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260620 00:19:28.383801 11657 leader_election.cc:302] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab8dfaac12fa413ea65468609ab2f605; no voters:
I20260620 00:19:28.384012 11657 leader_election.cc:288] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:19:28.384039 11662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:19:28.384280 11662 raft_consensus.cc:686] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [term 1 LEADER]: Becoming Leader. State: Replica: ab8dfaac12fa413ea65468609ab2f605, State: Running, Role: LEADER
I20260620 00:19:28.384375 11657 sys_catalog.cc:564] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:19:28.384408 11662 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8dfaac12fa413ea65468609ab2f605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } }
I20260620 00:19:28.384850 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab8dfaac12fa413ea65468609ab2f605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8dfaac12fa413ea65468609ab2f605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } } }
I20260620 00:19:28.385030 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [sys.catalog]: This master's current role is: LEADER
I20260620 00:19:28.385205 11662 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab8dfaac12fa413ea65468609ab2f605. Latest consensus state: current_term: 1 leader_uuid: "ab8dfaac12fa413ea65468609ab2f605" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab8dfaac12fa413ea65468609ab2f605" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46509 } } }
I20260620 00:19:28.385309 11662 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605 [sys.catalog]: This master's current role is: LEADER
I20260620 00:19:28.385499 11671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:19:28.385819 11657 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 00:19:28.386117 11671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:19:28.389063 11679 mvcc.cc:205] Tried to move back new op lower bound from 7298722891313594368 to 7298722891303362560. Current Snapshot: MvccSnapshot[applied={T|T < 7298722891313594368}]
Jun 20 00:19:28 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914768, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:19:28.389848 11671 catalog_manager.cc:1347] Generated new cluster ID: a2c0179511144b2aaecbb085d73f3697
I20260620 00:19:28.389894 11671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:19:28.394837 11661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:28.382191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.0.98.62:46509)
0620 00:19:28.382360 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:28.382369 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:28.382400 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:19:28.382513 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:28.382521 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:28.382580 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:28.382729 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:19:28.383661 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:28.385000 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:28.386049 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:28.386699 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:28.387353 (+ 654us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:28.390196 (+ 2843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:19:28.390200 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:19:28.390200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:19:28.390210 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:19:28.392808 (+ 2598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:28.393214 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:28.393216 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:28.393217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:28.393325 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:28.393528 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:28.393528 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:28.393528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:28.393847 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:19:28.394415 (+ 568us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:28.394490 (+ 75us) server_negotiation.cc:299] Negotiation successful
0620 00:19:28.394612 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":75,"threads_started":1}
I20260620 00:19:28.407447 11671 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:19:28.408380 11671 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:19:28.411759 11671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605: Generated new TSK 0
I20260620 00:19:28.412048 11671 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:19:29.190974 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:19:29.279529 11702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:19:29.279739 11702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:19:29.279763 11702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:19:29.281092 11702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:19:29.281141 11702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:19:29.281169 11702 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:19:29.282684 11702 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:52187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11702
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:19:29.282935 11702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:19:29.283164 11702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:19:29.289367 11708 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:19:29.289420 11707 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:19:29.289781 11710 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:19:29.289904 11702 server_base.cc:1034] running on GCE node
I20260620 00:19:29.290508 11702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:19:29.290880 11702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:19:29.294229 11702 hybrid_clock.cc:648] HybridClock initialized: now 1781914769294196 us; error 41 us; skew 500 ppm
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:19:29.297716 11702 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:19:29.298280 11702 webserver.cc:460] Webserver started at http://127.0.98.1:40547/ using document root <none> and password file <none>
I20260620 00:19:29.298560 11702 fs_manager.cc:359] Metadata directory not provided
I20260620 00:19:29.298625 11702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:19:29.298763 11702 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:19:29.314152 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "c663a95c672c471da6f6d3e8a962018a"
format_stamp: "Formatted at 2026-06-20 00:19:29 on dist-test-slave-r81b"
server_key: "0548e47e775a22cbe3ec3d12608ecde7"
server_key_iv: "99cc4c8cf0bca2a4815f56f6171823f6"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:29.314795 11702 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c663a95c672c471da6f6d3e8a962018a"
format_stamp: "Formatted at 2026-06-20 00:19:29 on dist-test-slave-r81b"
server_key: "0548e47e775a22cbe3ec3d12608ecde7"
server_key_iv: "99cc4c8cf0bca2a4815f56f6171823f6"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:29.316848 11702 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 00:19:29.330888 11717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:29.331076 11702 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:19:29.331159 11702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "c663a95c672c471da6f6d3e8a962018a"
format_stamp: "Formatted at 2026-06-20 00:19:29 on dist-test-slave-r81b"
server_key: "0548e47e775a22cbe3ec3d12608ecde7"
server_key_iv: "99cc4c8cf0bca2a4815f56f6171823f6"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:29.331259 11702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:19:29.357317 11702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:19:29.360327 11702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:19:29.360539 11702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:19:29.360831 11702 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:19:29.361264 11702 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:19:29.361357 11702 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:29.361461 11702 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:19:29.361541 11702 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:29.368710 11702 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39209
I20260620 00:19:29.369097 11702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:19:29.376415 11832 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39209 every 8 connection(s)
I20260620 00:19:29.376962 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 11702
I20260620 00:19:29.377072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:19:29.377269 392 external_mini_cluster.cc:1428] Setting key 2f62ce545d7008e1c9c617384aa4e7cd
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:19:29.396063 11836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.380269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.1:35081)
0620 00:19:29.381735 (+ 1466us) negotiation.cc:107] Waiting for socket to connect
0620 00:19:29.381790 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0620 00:19:29.382545 (+ 755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:19:29.382993 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:19:29.382998 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:19:29.383277 (+ 279us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:19:29.383603 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.383609 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.384795 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.384796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.386286 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.386289 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.386859 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.386860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.387006 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.387657 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:19:29.387668 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:19:29.391922 (+ 4254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:19:29.394174 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:19:29.394182 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:19:29.394185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:19:29.394527 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:19:29.394737 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:19:29.394738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:19:29.394738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:19:29.394881 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:19:29.395181 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:19:29.395187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:19:29.395457 (+ 270us) client_negotiation.cc:769] Sending connection context
0620 00:19:29.395588 (+ 131us) client_negotiation.cc:240] Negotiation successful
0620 00:19:29.395781 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":92,"threads_started":1}
I20260620 00:19:29.396533 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.380557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35081 (local address 127.0.98.62:46509)
0620 00:19:29.382735 (+ 2178us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:29.382738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:29.382746 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:19:29.382770 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:29.382771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:29.382783 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:29.382845 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:19:29.383717 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.384681 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.386398 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.386720 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.386785 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.392038 (+ 5253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:19:29.392040 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:19:29.392041 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:19:29.392050 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:19:29.394073 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:29.394611 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:29.394612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:29.394613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:29.394675 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:29.394948 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:29.394949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:29.394950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:29.395099 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:19:29.395156 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:29.396302 (+ 1146us) server_negotiation.cc:299] Negotiation successful
0620 00:19:29.396408 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2145,"thread_start_us":81,"threads_started":1}
I20260620 00:19:29.397185 11833 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46509
I20260620 00:19:29.397305 11833 heartbeater.cc:464] Registering TS with master...
I20260620 00:19:29.397490 11833 heartbeater.cc:511] Master 127.0.98.62:46509 requested a full tablet report, sending...
I20260620 00:19:29.398073 11621 ts_manager.cc:194] Registered new tserver with Master: c663a95c672c471da6f6d3e8a962018a (127.0.98.1:39209)
I20260620 00:19:29.399037 11621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35081
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:19:29.412393 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:19:29.529716 11839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:19:29.530043 11839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:19:29.530088 11839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:19:29.531983 11839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:19:29.532056 11839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:19:29.532104 11839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:19:29.534493 11839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:52187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:19:29.534788 11839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:19:29.535132 11839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:19:29.542001 11849 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:19:29.542084 11847 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:19:29.542224 11846 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:19:29.543232 11839 server_base.cc:1034] running on GCE node
I20260620 00:19:29.543499 11839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:19:29.543774 11839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:19:29.544930 11839 hybrid_clock.cc:648] HybridClock initialized: now 1781914769544908 us; error 41 us; skew 500 ppm
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:19:29.547818 11839 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:19:29.548492 11839 webserver.cc:460] Webserver started at http://127.0.98.2:43269/ using document root <none> and password file <none>
I20260620 00:19:29.548746 11839 fs_manager.cc:359] Metadata directory not provided
I20260620 00:19:29.548810 11839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:19:29.548951 11839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:19:29.561861 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "0abde7e9c5344e4898b263492b22f7c0"
format_stamp: "Formatted at 2026-06-20 00:19:29 on dist-test-slave-r81b"
server_key: "439c1d89da1b473ef933b11ef7c978b7"
server_key_iv: "56865a27b0fc540d7a3e0fa2d0309e3c"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:29.562353 11839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0abde7e9c5344e4898b263492b22f7c0"
format_stamp: "Formatted at 2026-06-20 00:19:29 on dist-test-slave-r81b"
server_key: "439c1d89da1b473ef933b11ef7c978b7"
server_key_iv: "56865a27b0fc540d7a3e0fa2d0309e3c"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:29.564121 11839 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.001s
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:19:29.583276 11856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:29.583506 11839 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.000s sys 0.001s
I20260620 00:19:29.583635 11839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "0abde7e9c5344e4898b263492b22f7c0"
format_stamp: "Formatted at 2026-06-20 00:19:29 on dist-test-slave-r81b"
server_key: "439c1d89da1b473ef933b11ef7c978b7"
server_key_iv: "56865a27b0fc540d7a3e0fa2d0309e3c"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:29.583794 11839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:19:29.599179 11839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:19:29.601886 11839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:19:29.602110 11839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:19:29.602406 11839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:19:29.602826 11839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:19:29.602927 11839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:29.603003 11839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:19:29.603065 11839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:29.610724 11839 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42985
I20260620 00:19:29.611068 11969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42985 every 8 connection(s)
I20260620 00:19:29.611308 11839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:19:29.618759 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 11839
I20260620 00:19:29.618888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:19:29.619115 392 external_mini_cluster.cc:1428] Setting key 69b637a3f0316d14d3199b34dde3529d
I20260620 00:19:29.623663 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.612173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59867 (local address 127.0.98.62:46509)
0620 00:19:29.612228 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:29.612230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:29.612529 (+ 299us) server_negotiation.cc:407] Connection header received
0620 00:19:29.613001 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:29.613002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:29.613015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:29.613105 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:19:29.613646 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.614463 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.615499 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.615848 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.615924 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.619362 (+ 3438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:19:29.619364 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:19:29.619364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:19:29.619373 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:19:29.621273 (+ 1900us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:29.622309 (+ 1036us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:29.622311 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:29.622311 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:29.622387 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:29.622733 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:29.622735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:29.622735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:29.622910 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:19:29.622992 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:29.623464 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 00:19:29.623568 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:19:29.623878 11972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.612133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.2:59867)
0620 00:19:29.612405 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0620 00:19:29.612442 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:19:29.612906 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:19:29.613162 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:19:29.613165 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:19:29.613359 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:19:29.613551 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.613555 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.614565 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.614565 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.615403 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.615405 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.615953 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.615954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.616073 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.616771 (+ 698us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:19:29.616782 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:19:29.619219 (+ 2437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:19:29.621830 (+ 2611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:19:29.621839 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:19:29.621842 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:19:29.622204 (+ 362us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:19:29.622486 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:19:29.622487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:19:29.622487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:19:29.622651 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:19:29.622992 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:19:29.623004 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:19:29.623288 (+ 284us) client_negotiation.cc:769] Sending connection context
0620 00:19:29.623443 (+ 155us) client_negotiation.cc:240] Negotiation successful
0620 00:19:29.623611 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":72,"threads_started":1}
I20260620 00:19:29.624547 11970 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46509
I20260620 00:19:29.624660 11970 heartbeater.cc:464] Registering TS with master...
I20260620 00:19:29.624832 11970 heartbeater.cc:511] Master 127.0.98.62:46509 requested a full tablet report, sending...
I20260620 00:19:29.625191 11621 ts_manager.cc:194] Registered new tserver with Master: 0abde7e9c5344e4898b263492b22f7c0 (127.0.98.2:42985)
I20260620 00:19:29.625573 11621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:59867
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:19:29.647552 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:52187/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:19:29.774308 11975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:19:29.774693 11975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:19:29.774791 11975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:19:29.776785 11975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:19:29.776857 11975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:19:29.776913 11975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:19:29.779318 11975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:52187/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:19:29.779814 11975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:19:29.780162 11975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:19:29.786767 11982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:19:29.787176 11981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:19:29.787252 11984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:19:29.787588 11975 server_base.cc:1034] running on GCE node
I20260620 00:19:29.787828 11975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:19:29.788154 11975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:19:29.789336 11975 hybrid_clock.cc:648] HybridClock initialized: now 1781914769789298 us; error 54 us; skew 500 ppm
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:19:29.792753 11975 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:19:29.793372 11975 webserver.cc:460] Webserver started at http://127.0.98.3:44417/ using document root <none> and password file <none>
I20260620 00:19:29.793641 11975 fs_manager.cc:359] Metadata directory not provided
I20260620 00:19:29.793704 11975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:19:29.793920 11975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:19:29.811928 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "f2bc8ea8fd51409f8792f9fc618819ed"
format_stamp: "Formatted at 2026-06-20 00:19:29 on dist-test-slave-r81b"
server_key: "353f79f58c9fa6751762289b722c4917"
server_key_iv: "f543a6a7aa507bd534183dd0aff5c2f6"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:29.814359 11975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "f2bc8ea8fd51409f8792f9fc618819ed"
format_stamp: "Formatted at 2026-06-20 00:19:29 on dist-test-slave-r81b"
server_key: "353f79f58c9fa6751762289b722c4917"
server_key_iv: "f543a6a7aa507bd534183dd0aff5c2f6"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:29.820612 11975 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.000s
I20260620 00:19:29.840283 11991 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:29.841310 11975 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260620 00:19:29.841521 11975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "f2bc8ea8fd51409f8792f9fc618819ed"
format_stamp: "Formatted at 2026-06-20 00:19:29 on dist-test-slave-r81b"
server_key: "353f79f58c9fa6751762289b722c4917"
server_key_iv: "f543a6a7aa507bd534183dd0aff5c2f6"
server_key_version: "kuduclusterkey@0"
I20260620 00:19:29.841715 11975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:19:29.866534 11975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:19:29.869128 11975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:19:29.869347 11975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:19:29.869683 11975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:19:29.870178 11975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:19:29.870270 11975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:29.870348 11975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:19:29.870426 11975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:19:29.877877 11975 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34713
I20260620 00:19:29.878273 11975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:19:29.884480 12104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34713 every 8 connection(s)
Jun 20 00:19:29 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:19:29.887014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 11975
I20260620 00:19:29.887131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:19:29.887369 392 external_mini_cluster.cc:1428] Setting key 1f1553dfa6b58c5f3d4802b15806633d
I20260620 00:19:29.889591 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.879149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47811 (local address 127.0.98.62:46509)
0620 00:19:29.879214 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:29.879217 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:29.879480 (+ 263us) server_negotiation.cc:407] Connection header received
0620 00:19:29.880338 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:29.880340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:29.880355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:29.880405 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:19:29.881286 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.882069 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.883076 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.883294 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.883386 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.886167 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:19:29.886169 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:19:29.886169 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:19:29.886177 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:19:29.888035 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:29.888584 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:29.888585 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:29.888585 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:29.888646 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:29.888926 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:29.888927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:29.888928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:29.889050 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:19:29.889122 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:29.889416 (+ 294us) server_negotiation.cc:299] Negotiation successful
0620 00:19:29.889499 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260620 00:19:29.890048 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.879149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46509 (local address 127.0.98.3:47811)
0620 00:19:29.879326 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:19:29.879365 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:19:29.880197 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:19:29.880623 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:19:29.880626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:19:29.880885 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:19:29.881172 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.881179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.882192 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.882193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.882985 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.882987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.883403 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.883404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.883540 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.884251 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:19:29.884261 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:19:29.886045 (+ 1784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:19:29.888120 (+ 2075us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:19:29.888128 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:19:29.888157 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:19:29.888495 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:19:29.888736 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:19:29.888736 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:19:29.888737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:19:29.888850 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:19:29.889113 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:19:29.889120 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:19:29.889275 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 00:19:29.889376 (+ 101us) client_negotiation.cc:240] Negotiation successful
0620 00:19:29.889537 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":95,"threads_started":1}
I20260620 00:19:29.890515 12105 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46509
I20260620 00:19:29.890621 12105 heartbeater.cc:464] Registering TS with master...
I20260620 00:19:29.890821 12105 heartbeater.cc:511] Master 127.0.98.62:46509 requested a full tablet report, sending...
I20260620 00:19:29.891220 11621 ts_manager.cc:194] Registered new tserver with Master: f2bc8ea8fd51409f8792f9fc618819ed (127.0.98.3:34713)
I20260620 00:19:29.891608 11621 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47811
I20260620 00:19:29.899454 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:19:29.907450 11837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.900956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37978 (local address 127.0.98.62:46509)
0620 00:19:29.901001 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:29.901004 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:29.901128 (+ 124us) server_negotiation.cc:407] Connection header received
0620 00:19:29.901200 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:29.901202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:29.901212 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:29.901263 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:19:29.901869 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.902376 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.903145 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.903348 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.903424 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.904752 (+ 1328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:19:29.904754 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:19:29.904754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:19:29.904763 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:19:29.906205 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:29.906601 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:29.906603 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:29.906603 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:29.906660 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:19:29.906915 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:19:29.906916 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:19:29.906917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:19:29.907036 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:19:29.907142 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:29.907257 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 00:19:29.907338 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:19:29.909224 11621 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:19:29.910934 11621 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:19:29.919122 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.913920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49394 (local address 127.0.98.3:34713)
0620 00:19:29.914072 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:29.914078 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:29.914309 (+ 231us) server_negotiation.cc:407] Connection header received
0620 00:19:29.914389 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:29.914392 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:29.914412 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:29.914510 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:19:29.914857 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.915610 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.917827 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.918352 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.918427 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.918486 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:29.919006 (+ 520us) server_negotiation.cc:299] Negotiation successful
0620 00:19:29.919049 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260620 00:19:29.919165 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.913586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34713 (local address 127.0.0.1:49394)
0620 00:19:29.914196 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0620 00:19:29.914236 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:19:29.914303 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:19:29.914620 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:19:29.914624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:19:29.914627 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:19:29.914730 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.914733 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.915706 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.915707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.917711 (+ 2004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.917715 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.918762 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.918763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.918873 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.918891 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 00:19:29.918980 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 00:19:29.919044 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"thread_start_us":96,"threads_started":1}
I20260620 00:19:29.919919 12020 tablet_service.cc:1467] Processing CreateTablet for tablet e6d0c0f74c0d4ce0bf47265f3624ccfa (DEFAULT_TABLE table=test-table [id=8f47059a6d3b4c63aeb831abf3064a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:19:29.920311 12020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d0c0f74c0d4ce0bf47265f3624ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:19:29.923650 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.913586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42985 (local address 127.0.0.1:56972)
0620 00:19:29.919401 (+ 5815us) negotiation.cc:107] Waiting for socket to connect
0620 00:19:29.919415 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:19:29.919460 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:19:29.919805 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:19:29.919806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:19:29.919808 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:19:29.919847 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.919849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.921151 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.921152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.922353 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.922356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.923425 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.923426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.923483 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.923486 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:19:29.923530 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:19:29.923566 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5789,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260620 00:19:29.923897 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.913906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56972 (local address 127.0.98.2:42985)
0620 00:19:29.914125 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:29.914129 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:29.919537 (+ 5408us) server_negotiation.cc:407] Connection header received
0620 00:19:29.919596 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:29.919600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:29.919621 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:29.919704 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:19:29.919929 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.921033 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.922475 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.923248 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.923317 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.923391 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:29.923773 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 00:19:29.923814 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":103,"threads_started":1}
I20260620 00:19:29.924654 12123 tablet_bootstrap.cc:492] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: Bootstrap starting.
I20260620 00:19:29.924804 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.913781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39209 (local address 127.0.0.1:50162)
0620 00:19:29.920213 (+ 6432us) negotiation.cc:107] Waiting for socket to connect
0620 00:19:29.920226 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:19:29.920277 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:19:29.921332 (+ 1055us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:19:29.921333 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:19:29.921335 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:19:29.921386 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.921387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.922616 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.922617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.923669 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:29.923671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.924591 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.924592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:29.924664 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.924668 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:19:29.924698 (+ 30us) client_negotiation.cc:240] Negotiation successful
0620 00:19:29.924740 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6412,"thread_start_us":89,"threads_started":1}
I20260620 00:19:29.924690 11896 tablet_service.cc:1467] Processing CreateTablet for tablet e6d0c0f74c0d4ce0bf47265f3624ccfa (DEFAULT_TABLE table=test-table [id=8f47059a6d3b4c63aeb831abf3064a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:19:29.925058 11896 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d0c0f74c0d4ce0bf47265f3624ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:19:29.925237 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:29.914025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.0.98.1:39209)
0620 00:19:29.921149 (+ 7124us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:29.921155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:29.921166 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:19:29.921213 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:29.921215 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:29.921232 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:29.921457 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:19:29.921521 (+ 64us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.922529 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.923768 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:29.924489 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:29.925032 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:29.925101 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:29.925128 (+ 27us) server_negotiation.cc:299] Negotiation successful
0620 00:19:29.925171 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7079,"thread_start_us":67,"threads_started":1}
I20260620 00:19:29.925634 12123 tablet_bootstrap.cc:654] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: Neither blocks nor log segments found. Creating new log.
I20260620 00:19:29.925935 12123 log.cc:824] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: Log is configured to *not* fsync() on all Append() calls
I20260620 00:19:29.926198 11761 tablet_service.cc:1467] Processing CreateTablet for tablet e6d0c0f74c0d4ce0bf47265f3624ccfa (DEFAULT_TABLE table=test-table [id=8f47059a6d3b4c63aeb831abf3064a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:19:29.926509 11761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6d0c0f74c0d4ce0bf47265f3624ccfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:19:29.930016 12125 tablet_bootstrap.cc:492] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a: Bootstrap starting.
I20260620 00:19:29.930707 12125 tablet_bootstrap.cc:654] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a: Neither blocks nor log segments found. Creating new log.
I20260620 00:19:29.931013 12125 log.cc:824] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a: Log is configured to *not* fsync() on all Append() calls
I20260620 00:19:29.931320 12126 tablet_bootstrap.cc:492] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: Bootstrap starting.
I20260620 00:19:29.932288 12126 tablet_bootstrap.cc:654] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: Neither blocks nor log segments found. Creating new log.
I20260620 00:19:29.932585 12126 log.cc:824] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:19:29.933388 12125 tablet_bootstrap.cc:492] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a: No bootstrap required, opened a new log
I20260620 00:19:29.933462 12125 ts_tablet_manager.cc:1397] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260620 00:19:29.936039 12125 raft_consensus.cc:348] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } }
I20260620 00:19:29.936249 12125 raft_consensus.cc:374] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:19:29.936290 12125 raft_consensus.cc:729] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c663a95c672c471da6f6d3e8a962018a, State: Initialized, Role: FOLLOWER
I20260620 00:19:29.936429 12125 consensus_queue.cc:260] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } }
I20260620 00:19:29.936695 12123 tablet_bootstrap.cc:492] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: No bootstrap required, opened a new log
I20260620 00:19:29.936702 12125 ts_tablet_manager.cc:1428] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:19:29.936787 12123 ts_tablet_manager.cc:1397] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260620 00:19:29.936820 11833 heartbeater.cc:503] Master 127.0.98.62:46509 was elected leader, sending a full tablet report...
I20260620 00:19:29.938639 12126 tablet_bootstrap.cc:492] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: No bootstrap required, opened a new log
I20260620 00:19:29.938702 12126 ts_tablet_manager.cc:1397] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.002s
I20260620 00:19:29.939307 12123 raft_consensus.cc:348] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } }
I20260620 00:19:29.939492 12123 raft_consensus.cc:374] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:19:29.939534 12123 raft_consensus.cc:729] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2bc8ea8fd51409f8792f9fc618819ed, State: Initialized, Role: FOLLOWER
I20260620 00:19:29.939661 12123 consensus_queue.cc:260] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } }
I20260620 00:19:29.939946 12123 ts_tablet_manager.cc:1428] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:19:29.939989 12105 heartbeater.cc:503] Master 127.0.98.62:46509 was elected leader, sending a full tablet report...
I20260620 00:19:29.940964 12126 raft_consensus.cc:348] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } }
I20260620 00:19:29.941093 12126 raft_consensus.cc:374] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:19:29.941114 12126 raft_consensus.cc:729] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0abde7e9c5344e4898b263492b22f7c0, State: Initialized, Role: FOLLOWER
I20260620 00:19:29.941257 12126 consensus_queue.cc:260] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } }
I20260620 00:19:29.941553 11970 heartbeater.cc:503] Master 127.0.98.62:46509 was elected leader, sending a full tablet report...
I20260620 00:19:29.941541 12126 ts_tablet_manager.cc:1428] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 00:19:30.047330 12130 raft_consensus.cc:482] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:19:30.047490 12130 raft_consensus.cc:504] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } }
I20260620 00:19:30.047930 12130 leader_election.cc:288] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c663a95c672c471da6f6d3e8a962018a (127.0.98.1:39209), 0abde7e9c5344e4898b263492b22f7c0 (127.0.98.2:42985)
I20260620 00:19:30.051920 12107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:30.048085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39209 (local address 127.0.98.3:36183)
0620 00:19:30.048204 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0620 00:19:30.048216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:19:30.048268 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:19:30.048598 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:19:30.048599 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:19:30.048601 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:19:30.048668 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:30.048670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.049583 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.049584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:30.050839 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:30.050842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.051671 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.051671 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:30.051744 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:30.051747 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:19:30.051793 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 00:19:30.051829 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"mutex_wait_us":42}
I20260620 00:19:30.051988 12121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:30.048263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41909 (local address 127.0.98.2:42985)
0620 00:19:30.048307 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:30.048310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:30.048562 (+ 252us) server_negotiation.cc:407] Connection header received
0620 00:19:30.048663 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:30.048665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:30.048676 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:30.048706 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:19:30.048938 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.049635 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.051036 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.051564 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.051617 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:30.051629 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:30.051858 (+ 229us) server_negotiation.cc:299] Negotiation successful
0620 00:19:30.051913 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:19:30.052249 12132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:30.048342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42985 (local address 127.0.98.3:41909)
0620 00:19:30.048498 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 00:19:30.048508 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:19:30.048587 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:19:30.048773 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:19:30.048773 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:19:30.048775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:19:30.048852 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:30.048853 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.049760 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.049761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:30.050930 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:19:30.050933 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.051691 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.051691 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:19:30.051771 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:30.051775 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:19:30.052098 (+ 323us) client_negotiation.cc:240] Negotiation successful
0620 00:19:30.052162 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260620 00:19:30.052599 12122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:30.048368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36183 (local address 127.0.98.1:39209)
0620 00:19:30.048412 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:30.048414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:30.048421 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:19:30.048447 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:30.048448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:30.048455 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:30.048499 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:19:30.048738 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.049490 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.050952 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.051567 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.051631 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:30.051640 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:30.052495 (+ 855us) server_negotiation.cc:299] Negotiation successful
0620 00:19:30.052535 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:19:30.052568 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d0c0f74c0d4ce0bf47265f3624ccfa" candidate_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0abde7e9c5344e4898b263492b22f7c0" is_pre_election: true
I20260620 00:19:30.052742 11924 raft_consensus.cc:2454] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2bc8ea8fd51409f8792f9fc618819ed in term 0.
I20260620 00:19:30.052850 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d0c0f74c0d4ce0bf47265f3624ccfa" candidate_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c663a95c672c471da6f6d3e8a962018a" is_pre_election: true
I20260620 00:19:30.052981 11782 raft_consensus.cc:2454] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f2bc8ea8fd51409f8792f9fc618819ed in term 0.
I20260620 00:19:30.052974 11993 leader_election.cc:302] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0abde7e9c5344e4898b263492b22f7c0, f2bc8ea8fd51409f8792f9fc618819ed; no voters:
I20260620 00:19:30.053100 12130 raft_consensus.cc:2781] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:19:30.053140 12130 raft_consensus.cc:482] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:19:30.053162 12130 raft_consensus.cc:3037] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:19:30.054126 12130 raft_consensus.cc:504] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } }
I20260620 00:19:30.054275 12130 leader_election.cc:288] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [CANDIDATE]: Term 1 election: Requested vote from peers c663a95c672c471da6f6d3e8a962018a (127.0.98.1:39209), 0abde7e9c5344e4898b263492b22f7c0 (127.0.98.2:42985)
I20260620 00:19:30.054410 11782 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d0c0f74c0d4ce0bf47265f3624ccfa" candidate_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c663a95c672c471da6f6d3e8a962018a"
I20260620 00:19:30.054484 11782 raft_consensus.cc:3037] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:19:30.054476 11924 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6d0c0f74c0d4ce0bf47265f3624ccfa" candidate_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0abde7e9c5344e4898b263492b22f7c0"
I20260620 00:19:30.054551 11924 raft_consensus.cc:3037] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:19:30.055367 11782 raft_consensus.cc:2454] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2bc8ea8fd51409f8792f9fc618819ed in term 1.
I20260620 00:19:30.055455 11924 raft_consensus.cc:2454] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f2bc8ea8fd51409f8792f9fc618819ed in term 1.
I20260620 00:19:30.055560 11995 leader_election.cc:302] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c663a95c672c471da6f6d3e8a962018a, f2bc8ea8fd51409f8792f9fc618819ed; no voters:
I20260620 00:19:30.055644 12130 raft_consensus.cc:2781] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:19:30.055779 12130 raft_consensus.cc:686] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 1 LEADER]: Becoming Leader. State: Replica: f2bc8ea8fd51409f8792f9fc618819ed, State: Running, Role: LEADER
I20260620 00:19:30.055886 12130 consensus_queue.cc:237] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } }
I20260620 00:19:30.056715 11621 catalog_manager.cc:5495] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed reported cstate change: term changed from 0 to 1, leader changed from <none> to f2bc8ea8fd51409f8792f9fc618819ed (127.0.98.3). New cstate: current_term: 1 leader_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f2bc8ea8fd51409f8792f9fc618819ed" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34713 } health_report { overall_health: HEALTHY } } }
I20260620 00:19:30.075629 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:19:30.071421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49400 (local address 127.0.98.3:34713)
0620 00:19:30.071473 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:19:30.071475 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:19:30.071485 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:19:30.071566 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:19:30.071568 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:19:30.071591 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:19:30.071676 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:19:30.072500 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.073237 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.074369 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:19:30.074652 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:19:30.074724 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:19:30.074990 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:19:30.075143 (+ 153us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:19:30.075229 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0620 00:19:30.075507 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 00:19:30.075539 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:19:30.080937 11782 raft_consensus.cc:1261] T e6d0c0f74c0d4ce0bf47265f3624ccfa P c663a95c672c471da6f6d3e8a962018a [term 1 FOLLOWER]: Refusing update from remote peer f2bc8ea8fd51409f8792f9fc618819ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:19:30.081360 12133 consensus_queue.cc:1035] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "c663a95c672c471da6f6d3e8a962018a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:19:30.080540 11924 raft_consensus.cc:1261] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [term 1 FOLLOWER]: Refusing update from remote peer f2bc8ea8fd51409f8792f9fc618819ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:19:30.082131 12133 consensus_queue.cc:1035] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "0abde7e9c5344e4898b263492b22f7c0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:19:30.089177 12136 mvcc.cc:205] Tried to move back new op lower bound from 7298722898245984256 to 7298722898149564416. Current Snapshot: MvccSnapshot[applied={T|T < 7298722898245984256}]
I20260620 00:19:30.089525 12137 mvcc.cc:205] Tried to move back new op lower bound from 7298722898245984256 to 7298722898149564416. Current Snapshot: MvccSnapshot[applied={T|T < 7298722898245984256}]
Jun 20 00:19:30 dist-test-slave-r81b krb5kdc[11145](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914769, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:19:30.092908 12139 mvcc.cc:205] Tried to move back new op lower bound from 7298722898245984256 to 7298722898149564416. Current Snapshot: MvccSnapshot[applied={T|T < 7298722898245984256}]
I20260620 00:19:30.096943 11621 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:19:30.097083 11621 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37978:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:19:30.098330 11621 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ab8dfaac12fa413ea65468609ab2f605: Sending DeleteTablet for 3 replicas of tablet e6d0c0f74c0d4ce0bf47265f3624ccfa
I20260620 00:19:30.098901 11896 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d0c0f74c0d4ce0bf47265f3624ccfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:19:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56972
I20260620 00:19:30.099398 11761 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d0c0f74c0d4ce0bf47265f3624ccfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:19:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50162
I20260620 00:19:30.099864 12020 tablet_service.cc:1514] Processing DeleteTablet for tablet e6d0c0f74c0d4ce0bf47265f3624ccfa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:19:30 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49394
I20260620 00:19:30.100533 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 11702
I20260620 00:19:30.103276 12147 tablet_replica.cc:330] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: stopping tablet replica
I20260620 00:19:30.103468 12147 raft_consensus.cc:2229] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:19:30.103629 12147 raft_consensus.cc:2258] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:19:30.104236 12147 ts_tablet_manager.cc:1905] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:19:30.106071 12147 ts_tablet_manager.cc:1918] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:19:30.106242 12147 log.cc:1196] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/e6d0c0f74c0d4ce0bf47265f3624ccfa
I20260620 00:19:30.106628 12147 ts_tablet_manager.cc:1939] T e6d0c0f74c0d4ce0bf47265f3624ccfa P f2bc8ea8fd51409f8792f9fc618819ed: Deleting consensus metadata
I20260620 00:19:30.107141 12145 tablet_replica.cc:330] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: stopping tablet replica
I20260620 00:19:30.107184 11605 catalog_manager.cc:4841] TS f2bc8ea8fd51409f8792f9fc618819ed (127.0.98.3:34713): tablet e6d0c0f74c0d4ce0bf47265f3624ccfa (table test-table [id=8f47059a6d3b4c63aeb831abf3064a58]) successfully deleted
I20260620 00:19:30.107285 12145 raft_consensus.cc:2229] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:19:30.107409 12145 raft_consensus.cc:2258] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:19:30.107969 12145 ts_tablet_manager.cc:1905] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:19:30.109606 12145 ts_tablet_manager.cc:1918] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:19:30.109751 12145 log.cc:1196] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/e6d0c0f74c0d4ce0bf47265f3624ccfa
I20260620 00:19:30.110051 12145 ts_tablet_manager.cc:1939] T e6d0c0f74c0d4ce0bf47265f3624ccfa P 0abde7e9c5344e4898b263492b22f7c0: Deleting consensus metadata
I20260620 00:19:30.110562 11606 catalog_manager.cc:4841] TS 0abde7e9c5344e4898b263492b22f7c0 (127.0.98.2:42985): tablet e6d0c0f74c0d4ce0bf47265f3624ccfa (table test-table [id=8f47059a6d3b4c63aeb831abf3064a58]) successfully deleted
W20260620 00:19:30.112457 11608 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39209 (error 108)
I20260620 00:19:30.113134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 11839
I20260620 00:19:30.120055 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 11975
I20260620 00:19:30.125756 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 11589
I20260620 00:19:30.133221 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 00:19:30.477903 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 00:19:30.478019 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 00:19:30.822777 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 00:19:30.823 UTC [11175] LOG: received smart shutdown request
2026-06-20 00:19:30.826 UTC [11175] LOG: background worker "logical replication launcher" (PID 11184) exited with exit code 1
2026-06-20 00:19:30.826 UTC [11179] LOG: shutting down
2026-06-20 00:19:31.008 UTC [11175] LOG: database system is shut down
2026-06-20T00:19:31Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (47657 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:19:31 dist-test-slave-r81b krb5kdc[12157](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:19:31 dist-test-slave-r81b krb5kdc[12157](info): set up 2 sockets
Jun 20 00:19:31 dist-test-slave-r81b krb5kdc[12157](info): commencing operation
krb5kdc: starting...
W20260620 00:19:32.164116 392 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:19:32 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914772, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:19:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:19:32Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260620 00:19:32.334635 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskbwaPxd/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/postgres -l logfile start
2026-06-20 00:19:34.910 UTC [12187] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-06-20 00:19:34.910 UTC [12187] LOG: listening on IPv4 address "127.0.98.20", port 41571
2026-06-20 00:19:34.914 UTC [12187] LOG: listening on Unix socket "/tmp/.s.PGSQL.41571"
2026-06-20 00:19:34.950 UTC [12190] LOG: database system was shut down at 2026-06-20 00:19:33 UTC
2026-06-20 00:19:34.954 UTC [12187] LOG: database system is ready to accept connections
I20260620 00:19:35.904254 392 mini_postgres.cc:96] Postgres bound to 41571
2026-06-20 00:19:35.910 UTC [12198] FATAL: database "slave" does not exist
127.0.98.20:41571 - accepting connections
I20260620 00:19:35.911126 392 mini_ranger.cc:161] Starting Ranger...
I20260620 00:19:35.927910 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260620 00:19:36.480499 392 mini_ranger.cc:87] Created ranger Postgres database
I20260620 00:19:36.480588 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-admin
2026-06-20 00:19:36,766 [I] DB FLAVOR :POSTGRES
2026-06-20 00:19:36,766 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:19:36,766 [I] Checking connection..
2026-06-20 00:19:36,766 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-06-20 00:19:37,039 [I] Checking connection passed.
2026-06-20 00:19:37,039 [I] --------- Verifying version history table ---------
2026-06-20 00:19:37,040 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:19:37,283 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:19:37,284 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:19:37,533 [I] Table x_db_version_h does not exist in database ranger
2026-06-20 00:19:37,533 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-06-20 00:19:37,534 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-06-20 00:19:37.815 UTC [12262] WARNING: there is no transaction in progress
2026-06-20 00:19:37,822 [I] create_dbversion_catalog.sql file imported successfully
2026-06-20 00:19:37,822 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-06-20 00:19:38,063 [I] Table x_db_version_h already exists in database 'ranger'
2026-06-20 00:19:38,064 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:19:38,064 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-06-20 00:19:38,298 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-06-20 00:19:38,564 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-r81b.c.gcp-upstream.internal','N') ;"
2026-06-20 00:19:38,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:19:39,067 [I] Table x_portal_user does not exist in database ranger
2026-06-20 00:19:39,067 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:19:39,311 [I] Table x_policy_ref_group does not exist in database ranger
2026-06-20 00:19:39,311 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:19:39,548 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:19:39,810 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-06-20 00:19:39,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-06-20 00:19:40.823 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:40.840 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:40.856 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:40.872 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:40.889 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:40.909 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:40.967 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:40.974 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:40.990 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:41.001 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:41.013 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:41.028 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:41.039 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:41.046 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:41.054 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:41.062 UTC [12370] WARNING: there is no transaction in progress
2026-06-20 00:19:41,760 [I] ranger_core_db_postgres.sql file imported successfully
2026-06-20 00:19:41,760 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-06-20 00:19:42,007 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-06-20 00:19:42,007 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-06-20 00:19:42,268 [I] Table x_portal_user already exists in database 'ranger'
2026-06-20 00:19:42,268 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-06-20 00:19:42,521 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-06-20 00:19:42,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-06-20 00:19:42,756 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-06-20 00:19:43,011 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-r81b.c.gcp-upstream.internal';"
2026-06-20 00:19:43,247 [I] CORE_DB_SCHEMA import status has been updated
2026-06-20 00:19:43,248 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-06-20 00:19:43,512 [I] DB_PATCHES have already been applied
I20260620 00:19:43.519088 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskbwaPxd/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260620 00:19:43.519161 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260620 00:19:43.522461 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:41791
Jun 20, 2026 12:19:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:19:43 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:19:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-admin/ews/webapp
Jun 20, 2026 12:19:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Jun 20, 2026 12:19:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Jun 20, 2026 12:19:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-admin/ews/webapp].
Jun 20, 2026 12:19:44 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260620 00:19:44.642297 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.731s user 0.000s sys 0.009s
Jun 20, 2026 12:20:04 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Jun 20, 2026 12:20:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Jun 20, 2026 12:20:04 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Jun 20, 2026 12:20:04 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Jun 20, 2026 12:20:05 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260620 00:20:05.855545 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Jun 20 00:20:05 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914805, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260620 00:20:05.939610 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260620 00:20:05.949708 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260620 00:20:06.881482 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260620 00:20:06.881574 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-kms
I20260620 00:20:06.881595 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:41571
2026-06-20 00:20:07,065 [I] DB FLAVOR :POSTGRES
2026-06-20 00:20:07,065 [I] --------- Verifying Ranger DB connection ---------
2026-06-20 00:20:07,065 [I] Checking connection
2026-06-20 00:20:07,065 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-06-20 00:20:07,320 [I] connection success
2026-06-20 00:20:07,320 [I] --------- Verifying Ranger DB tables ---------
2026-06-20 00:20:07,320 [I] Verifying table ranger_masterkey in database rangerkms
2026-06-20 00:20:07,320 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-06-20 00:20:07,583 [I] Table ranger_masterkey does not exist in database rangerkms
2026-06-20 00:20:07,583 [I] --------- Importing Ranger Core DB Schema ---------
2026-06-20 00:20:07,583 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-06-20 00:20:07,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:41571/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-06-20 00:20:07,852 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260620 00:20:08.392843 392 mini_ranger_kms.cc:310] Created kms service
I20260620 00:20:08.585994 392 mini_ranger_kms.cc:326] Created kudu user
I20260620 00:20:08.630086 392 mini_ranger_kms.cc:343] Created rangerkms user
Jun 20 00:20:08 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914805, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:09.038676 392 mini_ranger_kms.cc:384] Added ranger policy
I20260620 00:20:09.038818 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskbwaPxd/build/release/bin/postgresql.jar:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskbwaPxd/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260620 00:20:09.038837 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260620 00:20:09.042428 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12550
I20260620 00:20:09.042515 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:37987
Jun 20, 2026 12:20:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Jun 20, 2026 12:20:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Jun 20, 2026 12:20:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Jun 20, 2026 12:20:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Jun 20, 2026 12:20:10 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-kms/ews/webapp].
Jun 20, 2026 12:20:10 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260620 00:20:11.315964 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.376s user 0.006s sys 0.156s
I20260620 00:20:11.316073 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260620 00:20:11.316121 392 mini_ranger_kms.cc:401] 127.0.98.20:37987/kms/v1/keys
Jun 20 00:20:11 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914811, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Jun 20 00:20:11 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914811, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:20:12 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914811, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Jun 20 00:20:14 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:20:14.467137 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41939
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41939
--ranger_config_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37987/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:14.550778 12601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:14.551168 12601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:14.551263 12601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:14.552731 12601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:14.552802 12601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:14.552825 12601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:14.552850 12601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:14.552876 12601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:14.554862 12601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:37987/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41939
--ranger_config_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:14.555132 12601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:14.555403 12601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:14.561115 12607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:14.561173 12601 server_base.cc:1034] running on GCE node
W20260620 00:20:14.561111 12606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:14.561203 12609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:14.561710 12601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:14.562111 12601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:14.563357 12601 hybrid_clock.cc:648] HybridClock initialized: now 1781914814563336 us; error 38 us; skew 500 ppm
Jun 20 00:20:14 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:14.569808 12601 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:20:14.570302 12601 webserver.cc:460] Webserver started at http://127.0.98.62:36889/ using document root <none> and password file <none>
I20260620 00:20:14.570482 12601 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:14.570518 12601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:14.570604 12601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:20:14 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:14.623121 12601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "48e4c14a88004ae98c07e3408d05acf2"
format_stamp: "Formatted at 2026-06-20 00:20:14 on dist-test-slave-r81b"
server_key: "7b4683a717689628dd2e858ead631d87"
server_key_iv: "9dd57e1e4c493f29717c6916da1ae839"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:14.623594 12601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "48e4c14a88004ae98c07e3408d05acf2"
format_stamp: "Formatted at 2026-06-20 00:20:14 on dist-test-slave-r81b"
server_key: "7b4683a717689628dd2e858ead631d87"
server_key_iv: "9dd57e1e4c493f29717c6916da1ae839"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:14.625449 12601 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 00:20:14.640836 12616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:14.641026 12601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:20:14.641126 12601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "48e4c14a88004ae98c07e3408d05acf2"
format_stamp: "Formatted at 2026-06-20 00:20:14 on dist-test-slave-r81b"
server_key: "7b4683a717689628dd2e858ead631d87"
server_key_iv: "9dd57e1e4c493f29717c6916da1ae839"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:14.641237 12601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:14.649596 12601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:14.651798 12601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:14.651985 12601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:14.657228 12601 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41939
I20260620 00:20:14.657301 12668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41939 every 8 connection(s)
I20260620 00:20:14.657696 12601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:20:14.658413 12669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:14.660866 12669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Bootstrap starting.
I20260620 00:20:14.661455 12669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:14.661742 12669 log.cc:824] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:14.662408 12669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: No bootstrap required, opened a new log
I20260620 00:20:14.663265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 12601
I20260620 00:20:14.663380 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:20:14.663576 392 external_mini_cluster.cc:1428] Setting key 516ca98d3d42bc02f704afa4874937ad
I20260620 00:20:14.664934 12669 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } }
I20260620 00:20:14.665095 12669 raft_consensus.cc:374] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:14.665158 12669 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e4c14a88004ae98c07e3408d05acf2, State: Initialized, Role: FOLLOWER
I20260620 00:20:14.665336 12669 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } }
I20260620 00:20:14.665433 12669 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:20:14.665467 12669 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:20:14.665540 12669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:14.666242 12669 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } }
I20260620 00:20:14.666404 12669 leader_election.cc:302] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48e4c14a88004ae98c07e3408d05acf2; no voters:
I20260620 00:20:14.666615 12669 leader_election.cc:288] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:20:14.666641 12674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:20:14.666898 12674 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 1 LEADER]: Becoming Leader. State: Replica: 48e4c14a88004ae98c07e3408d05acf2, State: Running, Role: LEADER
I20260620 00:20:14.667012 12669 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:14.667025 12674 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } }
I20260620 00:20:14.667475 12674 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48e4c14a88004ae98c07e3408d05acf2. Latest consensus state: current_term: 1 leader_uuid: "48e4c14a88004ae98c07e3408d05acf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } } }
I20260620 00:20:14.668777 12674 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:14.668648 12669 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 00:20:14.669028 12689 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:20:14.669193 12676 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "48e4c14a88004ae98c07e3408d05acf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } } }
I20260620 00:20:14.669286 12676 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: This master's current role is: LEADER
Jun 20 00:20:14 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:14.672752 12689 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:20:14.676596 12689 catalog_manager.cc:1347] Generated new cluster ID: b3785c1b18a8495e92a24594bb65f7ad
I20260620 00:20:14.676649 12689 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:20:14.677757 12673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:14.664162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.0.98.62:41939)
0620 00:20:14.664354 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:14.664364 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:14.664400 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:20:14.664950 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:14.664961 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:14.665031 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:14.665183 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:14.666047 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:14.667540 (+ 1493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:14.668721 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:14.669365 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:14.669487 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:14.672520 (+ 3033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:14.672584 (+ 64us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:14.672585 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:14.672622 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:14.675709 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:14.676191 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:14.676194 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:14.676194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:14.676326 (+ 132us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:14.676537 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:14.676538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:14.676538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:14.676940 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:14.677111 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:14.677189 (+ 78us) server_negotiation.cc:299] Negotiation successful
0620 00:20:14.677442 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":80,"threads_started":1}
I20260620 00:20:14.698684 12689 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:20:14.699523 12689 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:20:14.712852 12689 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Generated new TSK 0
I20260620 00:20:14.713310 12689 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:20:15.411931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41939
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:15.528281 12714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:15.528645 12714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:15.528728 12714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:15.530591 12714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:15.530675 12714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:15.530769 12714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:20:15.532871 12714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:37987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:15.533419 12714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:15.533699 12714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:15.538902 12722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:15.538938 12721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:15.538988 12724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:15.539399 12714 server_base.cc:1034] running on GCE node
I20260620 00:20:15.539606 12714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:15.539884 12714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:15.541071 12714 hybrid_clock.cc:648] HybridClock initialized: now 1781914815541051 us; error 35 us; skew 500 ppm
Jun 20 00:20:15 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:15.543798 12714 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:20:15.544358 12714 webserver.cc:460] Webserver started at http://127.0.98.1:33985/ using document root <none> and password file <none>
I20260620 00:20:15.544631 12714 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:15.544709 12714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:15.544878 12714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:20:15 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:15.558770 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "72369f9c8775401693dff62d1939258c"
format_stamp: "Formatted at 2026-06-20 00:20:15 on dist-test-slave-r81b"
server_key: "55a541ca2c453d4f868636aa32dc0805"
server_key_iv: "c6d09774249fd794f925f54f5f2b1c38"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:15.559473 12714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "72369f9c8775401693dff62d1939258c"
format_stamp: "Formatted at 2026-06-20 00:20:15 on dist-test-slave-r81b"
server_key: "55a541ca2c453d4f868636aa32dc0805"
server_key_iv: "c6d09774249fd794f925f54f5f2b1c38"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:15.562194 12714 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 00:20:15.579463 12731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:15.579717 12714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260620 00:20:15.579836 12714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "72369f9c8775401693dff62d1939258c"
format_stamp: "Formatted at 2026-06-20 00:20:15 on dist-test-slave-r81b"
server_key: "55a541ca2c453d4f868636aa32dc0805"
server_key_iv: "c6d09774249fd794f925f54f5f2b1c38"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:15.579939 12714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:15.610458 12714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:15.613050 12714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:15.613273 12714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:15.613576 12714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:15.614001 12714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:15.614087 12714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:15.614187 12714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:15.614259 12714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:15.620545 12714 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43623
I20260620 00:20:15.621033 12714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:20:15.622556 12844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43623 every 8 connection(s)
I20260620 00:20:15.630625 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 12714
I20260620 00:20:15.630751 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:20:15.630956 392 external_mini_cluster.cc:1428] Setting key 7f8f6be0066f1765acac1c8018f6222f
Jun 20 00:20:15 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:15.635490 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:15.623325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57993 (local address 127.0.98.62:41939)
0620 00:20:15.625116 (+ 1791us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:15.625118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:15.625133 (+ 15us) server_negotiation.cc:407] Connection header received
0620 00:20:15.625543 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:15.625544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:15.625557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:15.625620 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:15.626412 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:15.627209 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:15.628796 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:15.629110 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:15.629154 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:15.631861 (+ 2707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:15.631862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:15.631863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:15.631870 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:15.633817 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:15.634327 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:15.634329 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:15.634329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:15.634395 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:15.634690 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:15.634691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:15.634691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:15.634843 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:15.634890 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:15.635291 (+ 401us) server_negotiation.cc:299] Negotiation successful
0620 00:20:15.635383 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1759,"thread_start_us":94,"threads_started":1}
I20260620 00:20:15.636059 12847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:15.623299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41939 (local address 127.0.98.1:57993)
0620 00:20:15.624659 (+ 1360us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:15.624711 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:15.625371 (+ 660us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:15.625750 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:15.625755 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:15.626017 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:15.626301 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:15.626307 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:15.627326 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:15.627327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:15.628681 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:15.628683 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:15.629258 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:15.629259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:15.629399 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:15.630039 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:15.630049 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:15.631747 (+ 1698us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:15.633918 (+ 2171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:15.633926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:15.633929 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:15.634236 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:15.634469 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:15.634470 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:15.634470 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:15.634612 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:15.634940 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:15.634946 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:15.635133 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 00:20:15.635599 (+ 466us) client_negotiation.cc:240] Negotiation successful
0620 00:20:15.635819 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"thread_start_us":87,"threads_started":1}
I20260620 00:20:15.636791 12845 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41939
I20260620 00:20:15.636890 12845 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:15.637084 12845 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
I20260620 00:20:15.637668 12633 ts_manager.cc:194] Registered new tserver with Master: 72369f9c8775401693dff62d1939258c (127.0.98.1:43623)
I20260620 00:20:15.638590 12633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57993
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:20:15.667727 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41939
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:20:15 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260620 00:20:15.788908 12853 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:15.789224 12853 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:15.789270 12853 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:15.791139 12853 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:15.791278 12853 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:15.791369 12853 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:20:15.793727 12853 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:37987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12853
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:15.794013 12853 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:15.794359 12853 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:15.800779 12860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:15.800930 12862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:15.800805 12859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:15.801528 12853 server_base.cc:1034] running on GCE node
I20260620 00:20:15.801743 12853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:15.802030 12853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:15.803218 12853 hybrid_clock.cc:648] HybridClock initialized: now 1781914815803196 us; error 35 us; skew 500 ppm
Jun 20 00:20:15 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:15.806556 12853 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:20:15.807143 12853 webserver.cc:460] Webserver started at http://127.0.98.2:39203/ using document root <none> and password file <none>
I20260620 00:20:15.807425 12853 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:15.807498 12853 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:15.807691 12853 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:20:15 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:15.824826 12853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "7c1cff26ba4646afacce8eb8d2d0d310"
format_stamp: "Formatted at 2026-06-20 00:20:15 on dist-test-slave-r81b"
server_key: "506dba7e1999f5ed260c9046edc7e966"
server_key_iv: "355efd896dbcdc82cf834d125b18c6d5"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:15.825297 12853 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7c1cff26ba4646afacce8eb8d2d0d310"
format_stamp: "Formatted at 2026-06-20 00:20:15 on dist-test-slave-r81b"
server_key: "506dba7e1999f5ed260c9046edc7e966"
server_key_iv: "355efd896dbcdc82cf834d125b18c6d5"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:15.827190 12853 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260620 00:20:15.846103 12869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:15.846415 12853 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.000s
I20260620 00:20:15.846545 12853 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "7c1cff26ba4646afacce8eb8d2d0d310"
format_stamp: "Formatted at 2026-06-20 00:20:15 on dist-test-slave-r81b"
server_key: "506dba7e1999f5ed260c9046edc7e966"
server_key_iv: "355efd896dbcdc82cf834d125b18c6d5"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:15.846668 12853 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:15.872129 12853 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:15.874794 12853 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:15.875023 12853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:15.875334 12853 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:15.875790 12853 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:15.875887 12853 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:15.875960 12853 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:15.876020 12853 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:15.883477 12853 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32913
I20260620 00:20:15.884037 12853 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:20:15.884394 12982 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32913 every 8 connection(s)
I20260620 00:20:15.885174 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 12853
I20260620 00:20:15.885279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:20:15.885504 392 external_mini_cluster.cc:1428] Setting key 7a47905433b3dfc70c26ba6cc7edc34c
Jun 20 00:20:15 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:15.896324 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:15.885458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45405 (local address 127.0.98.62:41939)
0620 00:20:15.885507 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:15.885509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:15.885705 (+ 196us) server_negotiation.cc:407] Connection header received
0620 00:20:15.886442 (+ 737us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:15.886444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:15.886456 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:15.886486 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:15.887343 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:15.888167 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:15.889682 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:15.890040 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:15.890086 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:15.892866 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:15.892868 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:15.892868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:15.892876 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:15.894629 (+ 1753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:15.895148 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:15.895150 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:15.895150 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:15.895210 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:15.895493 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:15.895494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:15.895494 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:15.895635 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:15.895677 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:15.896088 (+ 411us) server_negotiation.cc:299] Negotiation successful
0620 00:20:15.896207 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:20:15.897250 12985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:15.885180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41939 (local address 127.0.98.2:45405)
0620 00:20:15.885571 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:15.885618 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:15.886315 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:15.886669 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:15.886674 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:15.886936 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:15.887231 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:15.887237 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:15.888281 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:15.888282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:15.889578 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:15.889581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:15.890200 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:15.890201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:15.890343 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:15.890986 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:15.890997 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:15.892740 (+ 1743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:15.894731 (+ 1991us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:15.894738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:15.894741 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:15.895054 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:15.895281 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:15.895282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:15.895282 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:15.895416 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:15.895722 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:15.895728 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:15.895923 (+ 195us) client_negotiation.cc:769] Sending connection context
0620 00:20:15.896687 (+ 764us) client_negotiation.cc:240] Negotiation successful
0620 00:20:15.896869 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":79,"threads_started":1}
I20260620 00:20:15.898126 12983 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41939
I20260620 00:20:15.898222 12983 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:15.898411 12983 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
I20260620 00:20:15.898813 12633 ts_manager.cc:194] Registered new tserver with Master: 7c1cff26ba4646afacce8eb8d2d0d310 (127.0.98.2:32913)
I20260620 00:20:15.899302 12633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:45405
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:20:15.927585 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41939
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:16.036468 12988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:16.036710 12988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:16.036748 12988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:16.038089 12988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:16.038159 12988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:16.038205 12988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:20:16.040307 12988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:37987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:16.040863 12988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:16.041204 12988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:16.047735 12996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:16.047809 12994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:16.047735 12993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:16.048307 12988 server_base.cc:1034] running on GCE node
I20260620 00:20:16.048581 12988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:16.048861 12988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:16.050027 12988 hybrid_clock.cc:648] HybridClock initialized: now 1781914816050007 us; error 37 us; skew 500 ppm
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:16.053048 12988 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:20:16.053567 12988 webserver.cc:460] Webserver started at http://127.0.98.3:41351/ using document root <none> and password file <none>
I20260620 00:20:16.053828 12988 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:16.053897 12988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:16.054028 12988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:16.067179 12988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "91467bba4af2415eadba374025ac50ff"
format_stamp: "Formatted at 2026-06-20 00:20:16 on dist-test-slave-r81b"
server_key: "161e8f1f51f11e13aa57a3f3ae762bc1"
server_key_iv: "d417a7cd6a3f0b4e18344203d67dbbc7"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:16.067754 12988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "91467bba4af2415eadba374025ac50ff"
format_stamp: "Formatted at 2026-06-20 00:20:16 on dist-test-slave-r81b"
server_key: "161e8f1f51f11e13aa57a3f3ae762bc1"
server_key_iv: "d417a7cd6a3f0b4e18344203d67dbbc7"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:16.077317 12988 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.002s sys 0.000s
I20260620 00:20:16.091717 13003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.091977 12988 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:16.092201 12988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "91467bba4af2415eadba374025ac50ff"
format_stamp: "Formatted at 2026-06-20 00:20:16 on dist-test-slave-r81b"
server_key: "161e8f1f51f11e13aa57a3f3ae762bc1"
server_key_iv: "d417a7cd6a3f0b4e18344203d67dbbc7"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:16.092417 12988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:16.125550 12988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:16.128238 12988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:16.128470 12988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:16.128805 12988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:16.129280 12988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:16.129384 12988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.129436 12988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:16.129462 12988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.137112 12988 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43149
I20260620 00:20:16.137147 13116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43149 every 8 connection(s)
I20260620 00:20:16.137610 12988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:16.147794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 12988
I20260620 00:20:16.147914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:20:16.148180 392 external_mini_cluster.cc:1428] Setting key 3c34a5357bdb3439807d89d9845c01eb
I20260620 00:20:16.150894 12848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:16.138495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46695 (local address 127.0.98.62:41939)
0620 00:20:16.138545 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:16.138547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:16.140457 (+ 1910us) server_negotiation.cc:407] Connection header received
0620 00:20:16.141184 (+ 727us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:16.141186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:16.141199 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:16.141230 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:16.142117 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.142947 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.144552 (+ 1605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.144852 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.144890 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:16.147337 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:16.147338 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:16.147339 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:16.147347 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:16.149232 (+ 1885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.149672 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.149673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.149674 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.149738 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.150054 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.150055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.150055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.150206 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:16.150251 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:16.150706 (+ 455us) server_negotiation.cc:299] Negotiation successful
0620 00:20:16.150816 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:20:16.151490 13119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:16.138488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41939 (local address 127.0.98.3:46695)
0620 00:20:16.140280 (+ 1792us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:16.140337 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:16.141050 (+ 713us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:16.141438 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:16.141443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:16.141700 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:16.141994 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:16.142001 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.143079 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.143080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:16.144440 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:16.144443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.144982 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.144983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:16.145145 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:16.145745 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:16.145756 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:16.147210 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:16.149323 (+ 2113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:16.149330 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:16.149333 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:16.149576 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:16.149805 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:16.149806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:16.149806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:16.149947 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:16.150301 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:16.150309 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:16.150521 (+ 212us) client_negotiation.cc:769] Sending connection context
0620 00:20:16.151067 (+ 546us) client_negotiation.cc:240] Negotiation successful
0620 00:20:16.151300 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1742,"thread_start_us":78,"threads_started":1}
I20260620 00:20:16.151921 13117 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41939
I20260620 00:20:16.152048 13117 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.152325 13117 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
I20260620 00:20:16.152755 12633 ts_manager.cc:194] Registered new tserver with Master: 91467bba4af2415eadba374025ac50ff (127.0.98.3:43149)
I20260620 00:20:16.153239 12633 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:46695
I20260620 00:20:16.159993 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:20:16.160706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 12714
I20260620 00:20:16.166668 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 12853
I20260620 00:20:16.173384 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 12988
I20260620 00:20:16.179682 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 12601
I20260620 00:20:16.186969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41939
--webserver_interface=127.0.98.62
--webserver_port=36889
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41939
--ranger_config_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37987/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37987/kms with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:16.268602 13120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:16.268822 13120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:16.268846 13120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:16.269977 13120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:16.270017 13120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:16.270031 13120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:16.270043 13120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:16.270056 13120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:16.271700 13120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37987/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41939
--ranger_config_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=36889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:16.271903 13120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:16.272070 13120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:16.277194 13126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:16.277259 13125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:16.277309 13120 server_base.cc:1034] running on GCE node
W20260620 00:20:16.277263 13128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:16.277694 13120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:16.278004 13120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:16.279181 13120 hybrid_clock.cc:648] HybridClock initialized: now 1781914816279159 us; error 43 us; skew 500 ppm
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:16.281819 13120 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:20:16.282222 13120 webserver.cc:460] Webserver started at http://127.0.98.62:36889/ using document root <none> and password file <none>
I20260620 00:20:16.282418 13120 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:16.282467 13120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:16.330212 13120 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:16.331085 13136 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.331238 13120 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:16.331315 13120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "48e4c14a88004ae98c07e3408d05acf2"
format_stamp: "Formatted at 2026-06-20 00:20:14 on dist-test-slave-r81b"
server_key: "7b4683a717689628dd2e858ead631d87"
server_key_iv: "9dd57e1e4c493f29717c6916da1ae839"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:16.331781 13120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:16.355855 13120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:16.357630 13120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:16.357760 13120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:16.362046 13120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41939
I20260620 00:20:16.362087 13188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41939 every 8 connection(s)
I20260620 00:20:16.362463 13120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:20:16.362901 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 13120
I20260620 00:20:16.363206 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:43623
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41939
--webserver_port=33985
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37987/kms with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:20:16.363880 13189 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:20:16.364627 13189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Bootstrap starting.
I20260620 00:20:16.372732 13189 log.cc:824] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:16.374491 13189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:20:16.374786 13189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Bootstrap complete.
I20260620 00:20:16.377216 13189 raft_consensus.cc:348] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } }
I20260620 00:20:16.377485 13189 raft_consensus.cc:729] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e4c14a88004ae98c07e3408d05acf2, State: Initialized, Role: FOLLOWER
I20260620 00:20:16.377698 13189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } }
I20260620 00:20:16.377797 13189 raft_consensus.cc:388] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:20:16.377849 13189 raft_consensus.cc:482] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:20:16.377897 13189 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:20:16.378911 13189 raft_consensus.cc:504] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } }
I20260620 00:20:16.379033 13189 leader_election.cc:302] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 48e4c14a88004ae98c07e3408d05acf2; no voters:
I20260620 00:20:16.379221 13189 leader_election.cc:288] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:20:16.379299 13193 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:20:16.379505 13193 raft_consensus.cc:686] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [term 2 LEADER]: Becoming Leader. State: Replica: 48e4c14a88004ae98c07e3408d05acf2, State: Running, Role: LEADER
I20260620 00:20:16.379513 13189 sys_catalog.cc:564] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:16.379678 13193 consensus_queue.cc:237] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } }
I20260620 00:20:16.379971 13194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "48e4c14a88004ae98c07e3408d05acf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } } }
I20260620 00:20:16.380033 13194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:16.380317 13195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 48e4c14a88004ae98c07e3408d05acf2. Latest consensus state: current_term: 2 leader_uuid: "48e4c14a88004ae98c07e3408d05acf2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "48e4c14a88004ae98c07e3408d05acf2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41939 } } }
I20260620 00:20:16.380371 13195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2 [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:16.380606 13198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:20:16.381076 13198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:20:16.381192 13198 catalog_manager.cc:1259] Loaded cluster ID: b3785c1b18a8495e92a24594bb65f7ad
I20260620 00:20:16.381245 13198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:20:16.381486 13189 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260620 00:20:16.382474 13198 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:20:16.382650 13198 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Loaded TSK: 0
I20260620 00:20:16.382858 13198 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:20:16.452478 13191 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:16.452711 13191 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:16.452754 13191 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:16.453960 13191 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:16.454030 13191 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:16.454074 13191 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:20:16.455693 13191 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:43623
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=33985
--tserver_master_addrs=127.0.98.62:41939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13191
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:16.455971 13191 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:16.456316 13191 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:16.463146 13228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:16.463308 13191 server_base.cc:1034] running on GCE node
W20260620 00:20:16.463047 13226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:16.463043 13225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:16.463719 13191 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:16.464010 13191 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:16.465224 13191 hybrid_clock.cc:648] HybridClock initialized: now 1781914816465206 us; error 26 us; skew 500 ppm
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:16.467999 13191 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:20:16.468621 13191 webserver.cc:460] Webserver started at http://127.0.98.1:33985/ using document root <none> and password file <none>
I20260620 00:20:16.468897 13191 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:16.468981 13191 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:16.489888 13191 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:16.490842 13238 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.491402 13191 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:20:16.491513 13191 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "72369f9c8775401693dff62d1939258c"
format_stamp: "Formatted at 2026-06-20 00:20:15 on dist-test-slave-r81b"
server_key: "55a541ca2c453d4f868636aa32dc0805"
server_key_iv: "c6d09774249fd794f925f54f5f2b1c38"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:16.491902 13191 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:16.515080 13191 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:16.518395 13191 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:16.518632 13191 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:16.518941 13191 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:16.519354 13191 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:16.519414 13191 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.519479 13191 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:16.519513 13191 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.526147 13191 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43623
I20260620 00:20:16.526440 13351 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43623 every 8 connection(s)
I20260620 00:20:16.526733 13191 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:20:16.528718 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 13191
I20260620 00:20:16.529081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:32913
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41939
--webserver_port=39203
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37987/kms with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:16.558895 13354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:16.527545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41939 (local address 127.0.98.1:55715)
0620 00:20:16.528232 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:16.528288 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:16.531552 (+ 3264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:16.532464 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:16.532469 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:16.532738 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:16.533017 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:16.533023 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.535452 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.535454 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:16.545839 (+ 10385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:16.546199 (+ 360us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.547287 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.547288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:16.547581 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:16.548263 (+ 682us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:16.548273 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:16.553923 (+ 5650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:16.556999 (+ 3076us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:16.557008 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:16.557010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:16.557349 (+ 339us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:16.557630 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:16.557631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:16.557631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:16.557779 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:16.558252 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:16.558258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:16.558419 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 00:20:16.558510 (+ 91us) client_negotiation.cc:240] Negotiation successful
0620 00:20:16.558674 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":89,"threads_started":1}
I20260620 00:20:16.559751 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:16.527951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55715 (local address 127.0.98.62:41939)
0620 00:20:16.530862 (+ 2911us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:16.530874 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:16.530925 (+ 51us) server_negotiation.cc:407] Connection header received
0620 00:20:16.531948 (+ 1023us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:16.531956 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:16.532028 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:16.532176 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:16.533146 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.535315 (+ 2169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.546365 (+ 11050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.547060 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.547163 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:16.554087 (+ 6924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:16.554092 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:16.554093 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:16.554135 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:16.556884 (+ 2749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.557448 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.557452 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.557453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.557561 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.557848 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.557849 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.557849 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.558136 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:16.558210 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:16.559409 (+ 1199us) server_negotiation.cc:299] Negotiation successful
0620 00:20:16.559538 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2843,"thread_start_us":95,"threads_started":1}
I20260620 00:20:16.560950 13352 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41939
I20260620 00:20:16.561045 13352 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.561239 13352 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
W20260620 00:20:16.562098 13352 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41939 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 00:20:16.562162 13352 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.562234 13352 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
I20260620 00:20:16.562626 13352 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.562697 13352 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
W20260620 00:20:16.563045 13352 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:20:16.640679 13356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:16.641032 13356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:16.641095 13356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:16.642992 13356 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:16.643098 13356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:16.643172 13356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:20:16.645407 13356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:32913
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39203
--tserver_master_addrs=127.0.98.62:41939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13356
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:16.645689 13356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:16.645968 13356 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:16.651218 13362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:16.651257 13361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:16.651475 13364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:16.651976 13356 server_base.cc:1034] running on GCE node
I20260620 00:20:16.652287 13356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:16.652596 13356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:16.653805 13356 hybrid_clock.cc:648] HybridClock initialized: now 1781914816653807 us; error 44 us; skew 500 ppm
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:16.657145 13356 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:20:16.657775 13356 webserver.cc:460] Webserver started at http://127.0.98.2:39203/ using document root <none> and password file <none>
I20260620 00:20:16.658043 13356 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:16.658125 13356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:16.684342 13356 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260620 00:20:16.685367 13372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.685596 13356 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:16.685707 13356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "7c1cff26ba4646afacce8eb8d2d0d310"
format_stamp: "Formatted at 2026-06-20 00:20:15 on dist-test-slave-r81b"
server_key: "506dba7e1999f5ed260c9046edc7e966"
server_key_iv: "355efd896dbcdc82cf834d125b18c6d5"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:16.686136 13356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:16.709751 13356 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:16.712131 13356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:16.712414 13356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:16.712719 13356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:16.713143 13356 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:16.713198 13356 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.713235 13356 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:16.713265 13356 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.720589 13356 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:32913
I20260620 00:20:16.721129 13356 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:20:16.722054 13485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:32913 every 8 connection(s)
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:16.730778 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:16.721949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48383 (local address 127.0.98.62:41939)
0620 00:20:16.722000 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:16.722002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:16.722254 (+ 252us) server_negotiation.cc:407] Connection header received
0620 00:20:16.722855 (+ 601us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:16.722857 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:16.722874 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:16.722925 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:16.723549 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.724067 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.725179 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.725403 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.725493 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:16.727755 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:16.727757 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:16.727757 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:16.727766 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:16.729287 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.729712 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.729713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.729714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.729772 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.730047 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.730048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.730048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.730170 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:16.730250 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:16.730593 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 00:20:16.730685 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:20:16.730983 13488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:16.721970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41939 (local address 127.0.98.2:48383)
0620 00:20:16.722129 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:16.722173 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:16.722729 (+ 556us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:16.722997 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:16.723001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:16.723198 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:16.723450 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:16.723455 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.724199 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.724200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:16.725083 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:16.725085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.725505 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.725506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:16.725687 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:16.726269 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:16.726278 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:16.727653 (+ 1375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:16.729380 (+ 1727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:16.729387 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:16.729390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:16.729621 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:16.729841 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:16.729842 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:16.729842 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:16.729975 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:16.730241 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:16.730247 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:16.730440 (+ 193us) client_negotiation.cc:769] Sending connection context
0620 00:20:16.730559 (+ 119us) client_negotiation.cc:240] Negotiation successful
0620 00:20:16.730710 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":89,"threads_started":1}
I20260620 00:20:16.731084 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 13356
I20260620 00:20:16.731380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:37987/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43149
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41939
--webserver_port=41351
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:42935
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37987/kms with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:20:16.732549 13486 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41939
I20260620 00:20:16.732702 13486 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.732950 13486 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
W20260620 00:20:16.733721 13486 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41939 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 00:20:16.733808 13486 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.733918 13486 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
I20260620 00:20:16.734239 13486 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.734354 13486 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
W20260620 00:20:16.734705 13486 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:20:16.833647 13489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:16.833978 13489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:16.834018 13489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:16.835937 13489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:16.836087 13489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:16.836208 13489 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:20:16.838594 13489 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:42935
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:37987/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=41351
--tserver_master_addrs=127.0.98.62:41939
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:16.839149 13489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:16.839497 13489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:16.846168 13494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:16.846280 13497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:16.846187 13495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:16.847673 13489 server_base.cc:1034] running on GCE node
I20260620 00:20:16.847911 13489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:16.848250 13489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:16.849439 13489 hybrid_clock.cc:648] HybridClock initialized: now 1781914816849418 us; error 39 us; skew 500 ppm
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:16.852860 13489 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:20:16.853539 13489 webserver.cc:460] Webserver started at http://127.0.98.3:41351/ using document root <none> and password file <none>
I20260620 00:20:16.853826 13489 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:16.853914 13489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:16.876787 13489 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:16.880319 13506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.880632 13489 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260620 00:20:16.880733 13489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "91467bba4af2415eadba374025ac50ff"
format_stamp: "Formatted at 2026-06-20 00:20:16 on dist-test-slave-r81b"
server_key: "161e8f1f51f11e13aa57a3f3ae762bc1"
server_key_iv: "d417a7cd6a3f0b4e18344203d67dbbc7"
server_key_version: "kuduclusterkey@0"
I20260620 00:20:16.881209 13489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:16.895402 13489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:16.897658 13489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:16.897812 13489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:16.898052 13489 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:16.898500 13489 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:16.898557 13489 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.898620 13489 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:16.898648 13489 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:16.906359 13489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43149
I20260620 00:20:16.906901 13619 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43149 every 8 connection(s)
I20260620 00:20:16.906981 13489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:20:16.909221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 13489
Jun 20 00:20:16 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914816, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:16.918503 13622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:16.908210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41939 (local address 127.0.98.3:55207)
0620 00:20:16.908479 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:16.908529 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:16.909058 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:16.909455 (+ 397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:16.909459 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:16.909759 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:16.910076 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:16.910084 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.910857 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.910857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:16.911830 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:16.911832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.912439 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.912441 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:16.912587 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:16.913235 (+ 648us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:16.913246 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:16.914917 (+ 1671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:16.916820 (+ 1903us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:16.916828 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:16.916831 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:16.917138 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:16.917367 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:16.917367 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:16.917368 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:16.917498 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:16.917777 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:16.917782 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:16.917969 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 00:20:16.918080 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:20:16.918240 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":89,"threads_started":1}
I20260620 00:20:16.918942 13355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:16.908290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55207 (local address 127.0.98.62:41939)
0620 00:20:16.908342 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:16.908344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:16.908623 (+ 279us) server_negotiation.cc:407] Connection header received
0620 00:20:16.909185 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:16.909186 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:16.909201 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:16.909233 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:16.910195 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.910763 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.911927 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.912247 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.912342 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:16.915026 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:16.915027 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:16.915028 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:16.915036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:16.916722 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.917223 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.917225 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.917225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.917298 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.917566 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.917567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.917568 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.917693 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:16.917751 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:16.918772 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0620 00:20:16.918856 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:20:16.919641 13620 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41939
I20260620 00:20:16.919736 13620 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.919888 13620 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
W20260620 00:20:16.920660 13620 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41939 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260620 00:20:16.920738 13620 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.920804 13620 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
I20260620 00:20:16.921334 13620 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:16.921523 13620 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
W20260620 00:20:16.922041 13620 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:20:16.923437 13624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:16.910973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42042 (local address 127.0.98.62:41939)
0620 00:20:16.912131 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:16.912133 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:16.912140 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:20:16.912190 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:16.912190 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:16.912203 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:16.912298 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:16.913083 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.913911 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.915000 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:16.915272 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:16.915340 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:16.919089 (+ 3749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:16.919090 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:16.919090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:16.919097 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:16.921192 (+ 2095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.922572 (+ 1380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.922573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.922574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.922639 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:16.922870 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:16.922871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:16.922872 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:16.923003 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:16.923172 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:16.923240 (+ 68us) server_negotiation.cc:299] Negotiation successful
0620 00:20:16.923334 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1132,"thread_start_us":66,"threads_started":1}
Jun 20 00:20:17 dist-test-slave-r81b krb5kdc[12157](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1781914817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:17.563228 13352 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:17.563396 13352 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
I20260620 00:20:17.563738 13153 ts_manager.cc:194] Registered new tserver with Master: 72369f9c8775401693dff62d1939258c (127.0.98.1:43623)
I20260620 00:20:17.564796 13153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55715
I20260620 00:20:17.734946 13486 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:17.735119 13486 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
I20260620 00:20:17.735656 13153 ts_manager.cc:194] Registered new tserver with Master: 7c1cff26ba4646afacce8eb8d2d0d310 (127.0.98.2:32913)
I20260620 00:20:17.736207 13153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48383
Jun 20 00:20:17 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260620 00:20:17.922304 13620 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:17.922488 13620 heartbeater.cc:511] Master 127.0.98.62:41939 requested a full tablet report, sending...
I20260620 00:20:17.922930 13153 ts_manager.cc:194] Registered new tserver with Master: 91467bba4af2415eadba374025ac50ff (127.0.98.3:43149)
I20260620 00:20:17.923461 13153 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55207
I20260620 00:20:17.925900 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:20:17.936228 13641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:17.927157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.0.98.62:41939)
0620 00:20:17.927469 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:17.927471 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:17.927635 (+ 164us) server_negotiation.cc:407] Connection header received
0620 00:20:17.927661 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:17.927662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:17.927673 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:17.927727 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:17.928491 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.929281 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.930362 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.930689 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.930749 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:17.932404 (+ 1655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:17.932406 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:17.932406 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:17.932415 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:17.934447 (+ 2032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:17.934906 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:17.934907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:17.934907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:17.934968 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:17.935183 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:17.935184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:17.935184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:17.935319 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:17.935450 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:17.935984 (+ 534us) server_negotiation.cc:299] Negotiation successful
0620 00:20:17.936095 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":70,"threads_started":1}
I20260620 00:20:17.944777 13153 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42056:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:20:17.947216 13153 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:20:17.958175 13652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:17.950970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43623 (local address 127.0.0.1:36112)
0620 00:20:17.951608 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:17.951652 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:17.951711 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:17.954224 (+ 2513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:17.954225 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:17.954227 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:17.954269 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:17.954270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.955325 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.955327 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:17.956843 (+ 1516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:17.956846 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.957817 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.957818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:17.957909 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:17.957928 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 00:20:17.957982 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:20:17.958048 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"mutex_wait_us":111,"thread_start_us":70,"threads_started":1}
I20260620 00:20:17.958175 13648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:17.951138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32913 (local address 127.0.0.1:42704)
0620 00:20:17.951911 (+ 773us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:17.951919 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:17.951950 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:17.952288 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:17.952291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:17.952294 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:17.952374 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:17.952377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.955359 (+ 2982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.955359 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:17.956849 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:17.956851 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.957817 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.957818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:17.957909 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:17.957928 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 00:20:17.957982 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:20:17.958048 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":760,"mutex_wait_us":20,"thread_start_us":57,"threads_started":1}
I20260620 00:20:17.958972 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:17.951062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.0.98.1:43623)
0620 00:20:17.953564 (+ 2502us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:17.953569 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:17.953580 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:20:17.953614 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:17.953616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:17.953631 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:17.954337 (+ 706us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:17.954413 (+ 76us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.955216 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.956948 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.957707 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.958707 (+ 1000us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:17.958815 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:17.958845 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 00:20:17.958890 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2460,"thread_start_us":48,"threads_started":1}
I20260620 00:20:17.959681 13286 tablet_service.cc:1467] Processing CreateTablet for tablet b864db2fca3746ba8d4ed0a4e3265316 (DEFAULT_TABLE table=test-table [id=1b7330a59b804f5c996881de64e67208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:17.960063 13286 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b864db2fca3746ba8d4ed0a4e3265316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:17.963013 13654 tablet_bootstrap.cc:492] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c: Bootstrap starting.
I20260620 00:20:17.963883 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:17.951264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.98.2:32913)
0620 00:20:17.952073 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:17.952079 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:17.952089 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:20:17.952128 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:17.952130 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:17.952168 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:17.952460 (+ 292us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:17.952542 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.954131 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.956949 (+ 2818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.957707 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.958234 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:17.958913 (+ 679us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:17.958938 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 00:20:17.958980 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":54,"threads_started":1}
I20260620 00:20:17.963897 13654 tablet_bootstrap.cc:654] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:17.964257 13654 log.cc:824] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:17.965087 13654 tablet_bootstrap.cc:492] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c: No bootstrap required, opened a new log
I20260620 00:20:17.965152 13654 ts_tablet_manager.cc:1397] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:20:17.966195 13420 tablet_service.cc:1467] Processing CreateTablet for tablet b864db2fca3746ba8d4ed0a4e3265316 (DEFAULT_TABLE table=test-table [id=1b7330a59b804f5c996881de64e67208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:17.966585 13420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b864db2fca3746ba8d4ed0a4e3265316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:17.967803 13654 raft_consensus.cc:348] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } }
I20260620 00:20:17.967967 13654 raft_consensus.cc:374] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:17.967999 13654 raft_consensus.cc:729] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72369f9c8775401693dff62d1939258c, State: Initialized, Role: FOLLOWER
I20260620 00:20:17.968375 13654 consensus_queue.cc:260] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } }
I20260620 00:20:17.968699 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:17.950742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43149 (local address 127.0.0.1:42204)
0620 00:20:17.951290 (+ 548us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:17.951792 (+ 502us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:17.951823 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:17.963716 (+ 11893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:17.963717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:17.963720 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:17.963778 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:17.963781 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.965792 (+ 2011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.965793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:17.967006 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:17.967010 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.968417 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.968419 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:17.968495 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:17.968501 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:20:17.968535 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 00:20:17.968586 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":68,"threads_started":1}
I20260620 00:20:17.968813 13654 ts_tablet_manager.cc:1428] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 00:20:17.968945 13352 heartbeater.cc:503] Master 127.0.98.62:41939 was elected leader, sending a full tablet report...
I20260620 00:20:17.969331 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:17.950833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42204 (local address 127.0.98.3:43149)
0620 00:20:17.959206 (+ 8373us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:17.959212 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:17.959223 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:20:17.959258 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:17.959260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:17.959277 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:17.964199 (+ 4922us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:17.964291 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.965375 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.967127 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:17.968305 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:17.969068 (+ 763us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:17.969159 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:17.969189 (+ 30us) server_negotiation.cc:299] Negotiation successful
0620 00:20:17.969244 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8330,"thread_start_us":48,"threads_started":1}
I20260620 00:20:17.970005 13554 tablet_service.cc:1467] Processing CreateTablet for tablet b864db2fca3746ba8d4ed0a4e3265316 (DEFAULT_TABLE table=test-table [id=1b7330a59b804f5c996881de64e67208]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:17.970368 13554 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b864db2fca3746ba8d4ed0a4e3265316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:17.973882 13658 tablet_bootstrap.cc:492] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Bootstrap starting.
I20260620 00:20:17.974184 13657 tablet_bootstrap.cc:492] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Bootstrap starting.
I20260620 00:20:17.974629 13658 tablet_bootstrap.cc:654] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:17.975001 13658 log.cc:824] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:17.975028 13657 tablet_bootstrap.cc:654] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:17.975762 13657 log.cc:824] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:17.975826 13658 tablet_bootstrap.cc:492] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: No bootstrap required, opened a new log
I20260620 00:20:17.975884 13658 ts_tablet_manager.cc:1397] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:20:17.976861 13657 tablet_bootstrap.cc:492] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: No bootstrap required, opened a new log
I20260620 00:20:17.976931 13657 ts_tablet_manager.cc:1397] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:20:17.978622 13658 raft_consensus.cc:348] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } }
I20260620 00:20:17.978830 13658 raft_consensus.cc:374] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:17.978865 13658 raft_consensus.cc:729] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 91467bba4af2415eadba374025ac50ff, State: Initialized, Role: FOLLOWER
I20260620 00:20:17.979420 13658 consensus_queue.cc:260] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } }
I20260620 00:20:17.979709 13658 ts_tablet_manager.cc:1428] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 00:20:17.979874 13620 heartbeater.cc:503] Master 127.0.98.62:41939 was elected leader, sending a full tablet report...
I20260620 00:20:17.980384 13657 raft_consensus.cc:348] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } }
I20260620 00:20:17.980962 13657 raft_consensus.cc:374] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:17.981002 13657 raft_consensus.cc:729] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7c1cff26ba4646afacce8eb8d2d0d310, State: Initialized, Role: FOLLOWER
I20260620 00:20:17.981135 13657 consensus_queue.cc:260] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } }
I20260620 00:20:17.981411 13657 ts_tablet_manager.cc:1428] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 00:20:17.981455 13486 heartbeater.cc:503] Master 127.0.98.62:41939 was elected leader, sending a full tablet report...
W20260620 00:20:18.030421 13353 tablet.cc:2369] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c: Can't schedule compaction. Clean time has not been advanced past its initial value.
Jun 20 00:20:18 dist-test-slave-r81b krb5kdc[12157](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914817, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260620 00:20:18.158598 13621 tablet.cc:2369] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:20:18.222785 13487 tablet.cc:2369] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:20:18.279153 13656 raft_consensus.cc:482] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:18.279356 13656 raft_consensus.cc:504] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } }
I20260620 00:20:18.279781 13656 leader_election.cc:288] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 91467bba4af2415eadba374025ac50ff (127.0.98.3:43149), 7c1cff26ba4646afacce8eb8d2d0d310 (127.0.98.2:32913)
I20260620 00:20:18.284881 13649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:18.280584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59619 (local address 127.0.98.3:43149)
0620 00:20:18.280716 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:18.280718 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:18.280933 (+ 215us) server_negotiation.cc:407] Connection header received
0620 00:20:18.281048 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:18.281049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:18.281065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:18.281096 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:18.281356 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.282069 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.283567 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.284311 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.284419 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:18.284434 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:18.284755 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 00:20:18.284806 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260620 00:20:18.285390 13665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:18.280025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43149 (local address 127.0.98.1:59619)
0620 00:20:18.280862 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:18.280874 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:18.280965 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:18.281179 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:18.281180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:18.281182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:18.281275 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:18.281277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.282193 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.282194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:18.283458 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:18.283461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.284564 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.284565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:18.284647 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:18.284678 (+ 31us) client_negotiation.cc:769] Sending connection context
0620 00:20:18.285221 (+ 543us) client_negotiation.cc:240] Negotiation successful
0620 00:20:18.285274 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":817,"thread_start_us":86,"threads_started":1}
I20260620 00:20:18.285943 13564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b864db2fca3746ba8d4ed0a4e3265316" candidate_uuid: "72369f9c8775401693dff62d1939258c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91467bba4af2415eadba374025ac50ff" is_pre_election: true
I20260620 00:20:18.286120 13564 raft_consensus.cc:2454] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72369f9c8775401693dff62d1939258c in term 0.
I20260620 00:20:18.286403 13239 leader_election.cc:302] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72369f9c8775401693dff62d1939258c, 91467bba4af2415eadba374025ac50ff; no voters:
I20260620 00:20:18.286547 13656 raft_consensus.cc:2781] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:20:18.286592 13656 raft_consensus.cc:482] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:20:18.286613 13656 raft_consensus.cc:3037] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:18.287086 13653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:18.280664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50529 (local address 127.0.98.2:32913)
0620 00:20:18.280714 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:18.280717 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:18.283647 (+ 2930us) server_negotiation.cc:407] Connection header received
0620 00:20:18.283755 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:18.283756 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:18.283770 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:18.283800 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:18.284016 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.284737 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.286092 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.286672 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.286718 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:18.286729 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:18.286970 (+ 241us) server_negotiation.cc:299] Negotiation successful
0620 00:20:18.287015 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:20:18.287312 13666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:18.280291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:32913 (local address 127.0.98.1:50529)
0620 00:20:18.283550 (+ 3259us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:18.283564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:18.283678 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:18.283863 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:18.283864 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:18.283866 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:18.283940 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:18.283942 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.284860 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.284860 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:18.285984 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:18.285987 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.286808 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.286808 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:18.286891 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:18.286894 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:20:18.287191 (+ 297us) client_negotiation.cc:240] Negotiation successful
0620 00:20:18.287236 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3237,"thread_start_us":112,"threads_started":1}
I20260620 00:20:18.287586 13656 raft_consensus.cc:504] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } }
I20260620 00:20:18.287598 13440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b864db2fca3746ba8d4ed0a4e3265316" candidate_uuid: "72369f9c8775401693dff62d1939258c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" is_pre_election: true
I20260620 00:20:18.287734 13656 leader_election.cc:288] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [CANDIDATE]: Term 1 election: Requested vote from peers 91467bba4af2415eadba374025ac50ff (127.0.98.3:43149), 7c1cff26ba4646afacce8eb8d2d0d310 (127.0.98.2:32913)
I20260620 00:20:18.287745 13440 raft_consensus.cc:2454] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72369f9c8775401693dff62d1939258c in term 0.
I20260620 00:20:18.287930 13440 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b864db2fca3746ba8d4ed0a4e3265316" candidate_uuid: "72369f9c8775401693dff62d1939258c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7c1cff26ba4646afacce8eb8d2d0d310"
I20260620 00:20:18.287989 13440 raft_consensus.cc:3037] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:18.288554 13564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b864db2fca3746ba8d4ed0a4e3265316" candidate_uuid: "72369f9c8775401693dff62d1939258c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "91467bba4af2415eadba374025ac50ff"
I20260620 00:20:18.288640 13564 raft_consensus.cc:3037] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:18.288826 13440 raft_consensus.cc:2454] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72369f9c8775401693dff62d1939258c in term 1.
I20260620 00:20:18.289623 13564 raft_consensus.cc:2454] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72369f9c8775401693dff62d1939258c in term 1.
I20260620 00:20:18.289883 13240 leader_election.cc:302] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72369f9c8775401693dff62d1939258c, 7c1cff26ba4646afacce8eb8d2d0d310; no voters:
I20260620 00:20:18.290105 13656 raft_consensus.cc:2781] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:20:18.290266 13656 raft_consensus.cc:686] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [term 1 LEADER]: Becoming Leader. State: Replica: 72369f9c8775401693dff62d1939258c, State: Running, Role: LEADER
I20260620 00:20:18.290369 13656 consensus_queue.cc:237] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } }
I20260620 00:20:18.291203 13153 catalog_manager.cc:5495] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c reported cstate change: term changed from 0 to 1, leader changed from <none> to 72369f9c8775401693dff62d1939258c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "72369f9c8775401693dff62d1939258c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72369f9c8775401693dff62d1939258c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43623 } health_report { overall_health: HEALTHY } } }
I20260620 00:20:18.309957 13651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:18.306603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36114 (local address 127.0.98.1:43623)
0620 00:20:18.306647 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:18.306649 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:18.306658 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:20:18.306701 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:18.306701 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:18.306725 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:18.306803 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:20:18.306954 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.307702 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.308956 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:18.309265 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:18.309346 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:18.309543 (+ 197us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:20:18.309683 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:20:18.309771 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:18.309830 (+ 59us) server_negotiation.cc:299] Negotiation successful
0620 00:20:18.309856 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:20:18.312481 13440 raft_consensus.cc:1261] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 1 FOLLOWER]: Refusing update from remote peer 72369f9c8775401693dff62d1939258c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:18.312817 13564 raft_consensus.cc:1261] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 1 FOLLOWER]: Refusing update from remote peer 72369f9c8775401693dff62d1939258c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:18.313262 13656 consensus_queue.cc:1035] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [LEADER]: Connected to new peer: Peer: permanent_uuid: "91467bba4af2415eadba374025ac50ff" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:18.313622 13656 consensus_queue.cc:1035] T b864db2fca3746ba8d4ed0a4e3265316 P 72369f9c8775401693dff62d1939258c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7c1cff26ba4646afacce8eb8d2d0d310" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 32913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:18.320686 13152 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:20:18.320801 13152 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42056:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:20:18.321758 13152 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 48e4c14a88004ae98c07e3408d05acf2: Sending DeleteTablet for 3 replicas of tablet b864db2fca3746ba8d4ed0a4e3265316
I20260620 00:20:18.322227 13554 tablet_service.cc:1514] Processing DeleteTablet for tablet b864db2fca3746ba8d4ed0a4e3265316 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42204
I20260620 00:20:18.322463 13286 tablet_service.cc:1514] Processing DeleteTablet for tablet b864db2fca3746ba8d4ed0a4e3265316 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36112
I20260620 00:20:18.323076 13420 tablet_service.cc:1514] Processing DeleteTablet for tablet b864db2fca3746ba8d4ed0a4e3265316 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42704
I20260620 00:20:18.323657 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 13191
I20260620 00:20:18.325768 13678 tablet_replica.cc:330] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: stopping tablet replica
I20260620 00:20:18.325901 13678 raft_consensus.cc:2229] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:20:18.325991 13678 pending_rounds.cc:62] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Trying to abort 2 pending ops.
I20260620 00:20:18.326025 13678 pending_rounds.cc:68] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298723095804375040 op_type: WRITE_OP write_request { tablet_id: "b864db2fca3746ba8d4ed0a4e3265316" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\356\273\327\321\006\"8\n\ntest-admin\022*\n 1b7330a59b804f5c996881de64e67208\020\001\030\001 \001(\001" signature: "<redacted>""\027\003)4\213$\270\202\247\276\035\250\244\205\352\2278\224\263y$\214iZ\207\351\031\020\375\351\362\335\254\033\354\001\261\177*\311\004p\002\037%\304\352\332\364\354&\023+\254\241-T\223$T1\036N\335" signing_key_seq_num: 0 } } request_id { client_id: "78893c4da62e492b9e984767a0eac90b" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 00:20:18.326295 13678 pending_rounds.cc:68] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298723095717879808 op_type: NO_OP noop_request { }
I20260620 00:20:18.326380 13678 raft_consensus.cc:2866] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 00:20:18.326418 13678 raft_consensus.cc:2258] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:20:18.326598 13680 tablet_replica.cc:330] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: stopping tablet replica
I20260620 00:20:18.326716 13680 raft_consensus.cc:2229] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:20:18.326797 13680 pending_rounds.cc:62] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Trying to abort 2 pending ops.
I20260620 00:20:18.326833 13680 pending_rounds.cc:68] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7298723095804375040 op_type: WRITE_OP write_request { tablet_id: "b864db2fca3746ba8d4ed0a4e3265316" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "val" type: INT32 is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\003\00090\000\0001\324\000\000" indirect_data: "<redacted>""" } external_consistency_mode: CLIENT_PROPAGATED authz_token { token_data: "\010\356\273\327\321\006\"8\n\ntest-admin\022*\n 1b7330a59b804f5c996881de64e67208\020\001\030\001 \001(\001" signature: "<redacted>""\027\003)4\213$\270\202\247\276\035\250\244\205\352\2278\224\263y$\214iZ\207\351\031\020\375\351\362\335\254\033\354\001\261\177*\311\004p\002\037%\304\352\332\364\354&\023+\254\241-T\223$T1\036N\335" signing_key_seq_num: 0 } } request_id { client_id: "78893c4da62e492b9e984767a0eac90b" seq_no: 0 first_incomplete_seq_no: 0 attempt_no: 0 }
I20260620 00:20:18.327085 13680 pending_rounds.cc:68] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7298723095717879808 op_type: NO_OP noop_request { }
I20260620 00:20:18.327147 13680 raft_consensus.cc:2866] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260620 00:20:18.327186 13680 raft_consensus.cc:2258] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:20:18.327580 13678 ts_tablet_manager.cc:1905] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:20:18.328493 13680 ts_tablet_manager.cc:1905] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:20:18.330188 13680 ts_tablet_manager.cc:1918] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:20:18.329926 13678 ts_tablet_manager.cc:1918] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:20:18.331323 13680 log.cc:1196] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/b864db2fca3746ba8d4ed0a4e3265316
I20260620 00:20:18.331404 13678 log.cc:1196] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/b864db2fca3746ba8d4ed0a4e3265316
I20260620 00:20:18.331662 13680 ts_tablet_manager.cc:1939] T b864db2fca3746ba8d4ed0a4e3265316 P 7c1cff26ba4646afacce8eb8d2d0d310: Deleting consensus metadata
I20260620 00:20:18.331681 13678 ts_tablet_manager.cc:1939] T b864db2fca3746ba8d4ed0a4e3265316 P 91467bba4af2415eadba374025ac50ff: Deleting consensus metadata
I20260620 00:20:18.332204 13138 catalog_manager.cc:4841] TS 7c1cff26ba4646afacce8eb8d2d0d310 (127.0.98.2:32913): tablet b864db2fca3746ba8d4ed0a4e3265316 (table test-table [id=1b7330a59b804f5c996881de64e67208]) successfully deleted
I20260620 00:20:18.332244 13137 catalog_manager.cc:4841] TS 91467bba4af2415eadba374025ac50ff (127.0.98.3:43149): tablet b864db2fca3746ba8d4ed0a4e3265316 (table test-table [id=1b7330a59b804f5c996881de64e67208]) successfully deleted
W20260620 00:20:18.334218 13509 connection.cc:535] server connection from 127.0.98.1:59619 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 00:20:18.334561 13137 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43623 (error 108)
I20260620 00:20:18.335315 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 13356
I20260620 00:20:18.341796 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:18.341646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43623 (local address 127.0.0.1:36130)
0620 00:20:18.341696 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:18.341744 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43623: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 00:20:18.342118 13137 catalog_manager.cc:4584] TS 72369f9c8775401693dff62d1939258c (127.0.98.1:43623): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b864db2fca3746ba8d4ed0a4e3265316: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43623: connect: Connection refused (error 111)
I20260620 00:20:18.343724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 13489
I20260620 00:20:18.350296 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 13120
I20260620 00:20:18.360960 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260620 00:20:18.695835 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260620 00:20:18.695926 392 mini_ranger.cc:66] Stopping Ranger...
I20260620 00:20:19.033445 392 mini_ranger.cc:68] Stopped Ranger
2026-06-20 00:20:19.033 UTC [12187] LOG: received smart shutdown request
2026-06-20 00:20:19.037 UTC [12187] LOG: background worker "logical replication launcher" (PID 12196) exited with exit code 1
2026-06-20 00:20:19.037 UTC [12191] LOG: shutting down
2026-06-20 00:20:19.228 UTC [12187] LOG: database system is shut down
2026-06-20T00:20:19Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48220 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:20:19 dist-test-slave-r81b krb5kdc[13692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:20:19 dist-test-slave-r81b krb5kdc[13692](info): set up 2 sockets
Jun 20 00:20:19 dist-test-slave-r81b krb5kdc[13692](info): commencing operation
krb5kdc: starting...
W20260620 00:20:20.388368 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:20:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:20:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:20:20.507908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39739
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39739,127.0.98.61:40813,127.0.98.60:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:20.590436 13705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:20.590703 13705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:20.590742 13705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:20.591948 13705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:20.592021 13705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:20.592046 13705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:20.592074 13705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:20.592106 13705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:20.593693 13705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39739,127.0.98.61:40813,127.0.98.60:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:20.593973 13705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:20.594209 13705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:20.599287 13710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:20.599292 13713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:20.599246 13711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:20.599479 13705 server_base.cc:1034] running on GCE node
I20260620 00:20:20.599713 13705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:20.599979 13705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:20.601122 13705 hybrid_clock.cc:648] HybridClock initialized: now 1781914820601121 us; error 63 us; skew 500 ppm
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:20.607039 13705 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:20:20.607599 13705 webserver.cc:460] Webserver started at http://127.0.98.62:45821/ using document root <none> and password file <none>
I20260620 00:20:20.607831 13705 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:20.607905 13705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:20.608059 13705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:20.609182 13705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "95db5e0151b14547b2030d52c746c426"
format_stamp: "Formatted at 2026-06-20 00:20:20 on dist-test-slave-r81b"
server_key: "270d4d5d2fa267ad9ce607994e4dc4fd"
server_key_iv: "01c3f7dc3a89a411ce0dfaf9e4acd08f"
server_key_version: "encryptionkey@0"
I20260620 00:20:20.609560 13705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "95db5e0151b14547b2030d52c746c426"
format_stamp: "Formatted at 2026-06-20 00:20:20 on dist-test-slave-r81b"
server_key: "270d4d5d2fa267ad9ce607994e4dc4fd"
server_key_iv: "01c3f7dc3a89a411ce0dfaf9e4acd08f"
server_key_version: "encryptionkey@0"
I20260620 00:20:20.610953 13705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:20:20.611774 13720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:20.611968 13705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:20.612072 13705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "95db5e0151b14547b2030d52c746c426"
format_stamp: "Formatted at 2026-06-20 00:20:20 on dist-test-slave-r81b"
server_key: "270d4d5d2fa267ad9ce607994e4dc4fd"
server_key_iv: "01c3f7dc3a89a411ce0dfaf9e4acd08f"
server_key_version: "encryptionkey@0"
I20260620 00:20:20.612190 13705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:20.631807 13705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:20.634150 13705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:20.634322 13705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:20.638319 13705 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39739
I20260620 00:20:20.638355 13772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39739 every 8 connection(s)
I20260620 00:20:20.638732 13705 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:20:20.639803 13773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:20.641395 13773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:20:20.643023 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 13705
I20260620 00:20:20.643144 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:20:20.643312 392 external_mini_cluster.cc:1428] Setting key 0d27677705884d87b6cc2db36467eed7
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 00:20:20.651504 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.642257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59034 (local address 127.0.98.62:39739)
0620 00:20:20.642430 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.642434 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.642461 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:20:20.642805 (+ 344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.642807 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.642828 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.642960 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.643709 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.644589 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.645533 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.645854 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.645982 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.648096 (+ 2114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.648103 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.648103 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.648110 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.649921 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.650365 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.650369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.650370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.650482 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.650712 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.650713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.650714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.650935 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.651002 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.651169 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.651326 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":96,"threads_started":1}
I20260620 00:20:20.651510 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.642028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39739 (local address 127.0.0.1:59034)
0620 00:20:20.642194 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.642237 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:20.642670 (+ 433us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:20.642967 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:20.642974 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:20.643349 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:20.643591 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.643599 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.644710 (+ 1111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.644711 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.645437 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.645439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.645952 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.645953 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.646015 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.646590 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:20.646608 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:20.648004 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:20.650040 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.650046 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.650049 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.650247 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.650560 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.650561 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.650561 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.650641 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.650997 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:20.651001 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:20.651056 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 00:20:20.651129 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:20:20.651435 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":105,"thread_start_us":91,"threads_started":1}
I20260620 00:20:20.652444 13773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:20:20.652854 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.652711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40813 (local address 127.0.0.1:46392)
0620 00:20:20.652778 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.652829 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
W20260620 00:20:20.652997 13721 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
I20260620 00:20:20.653491 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.653394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40813 (local address 127.0.0.1:46398)
0620 00:20:20.653444 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.653468 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 00:20:20.653616 13773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40813: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 00:20:20.667912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40813
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39739,127.0.98.61:40813,127.0.98.60:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:20:20.702865 13773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } attempt: 1
I20260620 00:20:20.703398 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.703247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40813 (local address 127.0.0.1:46410)
0620 00:20:20.703311 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.703361 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260620 00:20:20.703846 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.703763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40813 (local address 127.0.0.1:46412)
0620 00:20:20.703794 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.703825 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260620 00:20:20.704015 13773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40813: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
W20260620 00:20:20.751273 13780 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:20.751469 13780 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:20.751493 13780 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:20.752668 13780 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:20.752707 13780 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:20.752722 13780 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:20.752735 13780 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:20.752748 13780 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:20.754244 13780 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39739,127.0.98.61:40813,127.0.98.60:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:40813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13780
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:20.754491 13780 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:20.754729 13780 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:20.759610 13785 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:20.759833 13780 server_base.cc:1034] running on GCE node
W20260620 00:20:20.759599 13788 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:20.759599 13786 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:20.760231 13780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:20.760535 13780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:20.761679 13780 hybrid_clock.cc:648] HybridClock initialized: now 1781914820761668 us; error 29 us; skew 500 ppm
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:20.764415 13780 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 00:20:20.764860 13780 webserver.cc:460] Webserver started at http://127.0.98.61:35419/ using document root <none> and password file <none>
I20260620 00:20:20.765053 13780 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:20.765094 13780 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:20.765192 13780 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:20.766201 13780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/data/instance:
uuid: "cd8b8e977a1f4f1086d38e150d9af01b"
format_stamp: "Formatted at 2026-06-20 00:20:20 on dist-test-slave-r81b"
server_key: "c49550ad706a0d954a6436ea8f685177"
server_key_iv: "9131c82dae029f763a2e0ea7ceede875"
server_key_version: "encryptionkey@0"
I20260620 00:20:20.766505 13780 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/wal/instance:
uuid: "cd8b8e977a1f4f1086d38e150d9af01b"
format_stamp: "Formatted at 2026-06-20 00:20:20 on dist-test-slave-r81b"
server_key: "c49550ad706a0d954a6436ea8f685177"
server_key_iv: "9131c82dae029f763a2e0ea7ceede875"
server_key_version: "encryptionkey@0"
I20260620 00:20:20.767839 13780 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:20:20.768679 13795 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:20.768898 13780 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:20:20.768992 13780 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/wal
uuid: "cd8b8e977a1f4f1086d38e150d9af01b"
format_stamp: "Formatted at 2026-06-20 00:20:20 on dist-test-slave-r81b"
server_key: "c49550ad706a0d954a6436ea8f685177"
server_key_iv: "9131c82dae029f763a2e0ea7ceede875"
server_key_version: "encryptionkey@0"
I20260620 00:20:20.769078 13780 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:20.772284 13773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } attempt: 2
I20260620 00:20:20.772783 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.772591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40813 (local address 127.0.0.1:46422)
0620 00:20:20.772689 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.772737 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260620 00:20:20.773144 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.773047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40813 (local address 127.0.0.1:46434)
0620 00:20:20.773093 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.773119 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:20:20.773263 13773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:40813: Network error: Client connection negotiation failed: client connection to 127.0.98.61:40813: connect: Connection refused (error 111)
I20260620 00:20:20.784111 13780 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:20.786123 13780 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:20.786295 13780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:20.790509 13780 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:40813
I20260620 00:20:20.790542 13847 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:40813 every 8 connection(s)
I20260620 00:20:20.790926 13780 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:20:20.791842 13848 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:20.793148 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 13780
I20260620 00:20:20.793305 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-1/wal/instance
I20260620 00:20:20.793521 392 external_mini_cluster.cc:1428] Setting key eebf7a875a4027bf604e1cc0a5427b5d
I20260620 00:20:20.793702 13848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 00:20:20.805007 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.794509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59038 (local address 127.0.98.62:39739)
0620 00:20:20.794582 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.794585 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.794822 (+ 237us) server_negotiation.cc:407] Connection header received
0620 00:20:20.795530 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.795531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.795546 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.795609 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.796538 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.797085 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.798705 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.798920 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.798983 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.801622 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.801624 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.801625 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.801634 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.803216 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.803830 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.803831 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.803832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.803901 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.804251 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.804252 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.804252 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.804391 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.804474 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.804856 (+ 382us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.804937 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260620 00:20:20.805281 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.794446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39739 (local address 127.0.0.1:59038)
0620 00:20:20.794645 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.794695 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:20.795409 (+ 714us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:20.795694 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:20.795699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:20.796101 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:20.796433 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.796440 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.797191 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.797192 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.798605 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.798609 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.799017 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.799018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.799180 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.799784 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:20.799795 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:20.801499 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:20.803353 (+ 1854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.803361 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.803363 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.803722 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.803985 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.803986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.803987 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.804187 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.804478 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:20.804486 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:20.804657 (+ 171us) client_negotiation.cc:769] Sending connection context
0620 00:20:20.804793 (+ 136us) client_negotiation.cc:240] Negotiation successful
0620 00:20:20.804986 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":87,"threads_started":1}
I20260620 00:20:20.805920 13848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:20:20.814733 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.806217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40813 (local address 127.0.0.1:46440)
0620 00:20:20.806306 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.806317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:20.806363 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:20.806693 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:20.806701 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:20.806852 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:20.806903 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.806905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.807946 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.807947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.808725 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.808728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.809499 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.809500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.809571 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.810120 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:20.810127 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:20.811622 (+ 1495us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:20.813568 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.813569 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.813570 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.813744 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.813966 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.813967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.813967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.814021 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.814359 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:20.814360 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:20.814399 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 00:20:20.814460 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:20:20.814587 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 00:20:20.814800 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.806272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46440 (local address 127.0.98.61:40813)
0620 00:20:20.806473 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.806479 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.806508 (+ 29us) server_negotiation.cc:407] Connection header received
0620 00:20:20.806537 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.806539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.806558 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.806658 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.806989 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.807852 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.808826 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.809407 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.809503 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.811751 (+ 2248us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.811760 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.811760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.811768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.813462 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.813839 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.813842 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.813843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.813898 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.814102 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.814102 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.814103 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.814289 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.814383 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.814497 (+ 114us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.814636 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":49,"server-negotiator.queue_time_us":160,"thread_start_us":100,"threads_started":1}
I20260620 00:20:20.815526 13848 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:20:20.815963 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.815796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57522)
0620 00:20:20.815855 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.815904 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:20:20.816227 13797 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
I20260620 00:20:20.816793 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.816692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57528)
0620 00:20:20.816747 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.816773 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260620 00:20:20.816951 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 00:20:20.823271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36223
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43151
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39739,127.0.98.61:40813,127.0.98.60:36223
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:20:20.864408 13773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } attempt: 3
I20260620 00:20:20.866225 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } attempt: 1
I20260620 00:20:20.866621 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.866484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57542)
0620 00:20:20.866539 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.866581 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:20:20.866962 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.866884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57552)
0620 00:20:20.866924 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.866943 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260620 00:20:20.867049 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:20:20.872360 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.864884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46444 (local address 127.0.98.61:40813)
0620 00:20:20.864951 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.864953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.864966 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:20:20.865043 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.865044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.865057 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.865127 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.865468 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.865967 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.866823 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.867235 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.867303 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.869198 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.869199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.869200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.869207 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.870959 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.871393 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.871394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.871394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.871457 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.871745 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.871746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.871746 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.871892 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.871981 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.872187 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.872272 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:20:20.872371 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.864807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40813 (local address 127.0.0.1:46444)
0620 00:20:20.864880 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.864917 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:20.864971 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:20.865133 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:20.865134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:20.865312 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:20.865373 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.865376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.866052 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.866053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.866744 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.866747 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.867325 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.867326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.867388 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.867836 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:20.867841 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:20.869117 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:20.871063 (+ 1946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.871064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.871065 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.871315 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.871554 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.871556 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.871556 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.871653 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.872007 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:20.872008 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:20.872063 (+ 55us) client_negotiation.cc:769] Sending connection context
0620 00:20:20.872190 (+ 127us) client_negotiation.cc:240] Negotiation successful
0620 00:20:20.872305 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 00:20:20.872979 13773 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:20:20.873292 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.873205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57562)
0620 00:20:20.873239 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.873270 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260620 00:20:20.873593 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.873534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57568)
0620 00:20:20.873555 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.873580 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260620 00:20:20.873692 13773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
I20260620 00:20:20.904875 13773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } attempt: 1
I20260620 00:20:20.905368 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.905229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57574)
0620 00:20:20.905294 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.905328 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 00:20:20.905709 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.905597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57582)
0620 00:20:20.905645 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.905680 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260620 00:20:20.905798 13773 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
W20260620 00:20:20.912606 13855 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:20.912846 13855 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:20.912889 13855 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:20.914070 13855 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:20.914136 13855 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:20.914162 13855 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:20.914194 13855 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:20.914222 13855 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:20.915890 13855 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43151
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39739,127.0.98.61:40813,127.0.98.60:36223
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36223
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13855
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:20.916219 13855 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:20.916463 13855 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:20.921514 13860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:20.921581 13863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:20.921514 13861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:20.921731 13855 server_base.cc:1034] running on GCE node
I20260620 00:20:20.922037 13855 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:20.922303 13855 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:20.923506 13855 hybrid_clock.cc:648] HybridClock initialized: now 1781914820923479 us; error 40 us; skew 500 ppm
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:20.926182 13855 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 00:20:20.926587 13855 webserver.cc:460] Webserver started at http://127.0.98.60:36489/ using document root <none> and password file <none>
I20260620 00:20:20.926795 13855 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:20.926836 13855 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:20.926930 13855 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:20.928069 13855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/data/instance:
uuid: "8bf2700690664a7d88d79e6613598a5e"
format_stamp: "Formatted at 2026-06-20 00:20:20 on dist-test-slave-r81b"
server_key: "3f43acd8d74f93b5138ff970d0d0f511"
server_key_iv: "46fca0af240df472c5d9fe6048cbc79a"
server_key_version: "encryptionkey@0"
I20260620 00:20:20.928443 13855 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/wal/instance:
uuid: "8bf2700690664a7d88d79e6613598a5e"
format_stamp: "Formatted at 2026-06-20 00:20:20 on dist-test-slave-r81b"
server_key: "3f43acd8d74f93b5138ff970d0d0f511"
server_key_iv: "46fca0af240df472c5d9fe6048cbc79a"
server_key_version: "encryptionkey@0"
I20260620 00:20:20.929693 13855 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:20.930409 13870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:20.930559 13855 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:20:20.930647 13855 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/wal
uuid: "8bf2700690664a7d88d79e6613598a5e"
format_stamp: "Formatted at 2026-06-20 00:20:20 on dist-test-slave-r81b"
server_key: "3f43acd8d74f93b5138ff970d0d0f511"
server_key_iv: "46fca0af240df472c5d9fe6048cbc79a"
server_key_version: "encryptionkey@0"
I20260620 00:20:20.930704 13855 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:20.935225 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } attempt: 2
I20260620 00:20:20.935710 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.935580 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57598)
0620 00:20:20.935639 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.935681 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:20:20.936100 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.935996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57610)
0620 00:20:20.936046 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.936076 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260620 00:20:20.936223 13848 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:36223: Network error: Client connection negotiation failed: client connection to 127.0.98.60:36223: connect: Connection refused (error 111)
I20260620 00:20:20.959684 13855 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:20.961966 13855 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:20.962105 13855 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:20.966267 13855 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36223
I20260620 00:20:20.966312 13922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36223 every 8 connection(s)
I20260620 00:20:20.966681 13855 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:20:20.967566 13923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:20.968652 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 13855
I20260620 00:20:20.968793 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACert.1781914666918284-392-0/minicluster-data/master-2/wal/instance
I20260620 00:20:20.968989 392 external_mini_cluster.cc:1428] Setting key 156986f2fd65b99f39a5d35afafadf3b
I20260620 00:20:20.969566 13923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:20:20.978938 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.970330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59048 (local address 127.0.98.62:39739)
0620 00:20:20.970380 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.970382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.970527 (+ 145us) server_negotiation.cc:407] Connection header received
0620 00:20:20.970561 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.970562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.970573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.970643 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.971300 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.971860 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.972919 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.973204 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.973268 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.975830 (+ 2562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.975831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.975832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.975840 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.977433 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.977925 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.977927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.977927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.977994 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.978267 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.978268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.978268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.978394 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.978471 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.978749 (+ 278us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.978841 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:20:20.979341 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.970221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39739 (local address 127.0.0.1:59048)
0620 00:20:20.970384 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.970422 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:20.970510 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:20.970659 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:20.970662 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:20.970932 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:20.971172 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.971179 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.971917 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.971918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.972810 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.972812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.973292 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.973293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.973420 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.974035 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:20.974043 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:20.975730 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:20.977541 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.977549 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.977552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.977809 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.978065 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.978065 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.978066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.978190 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.978476 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:20.978483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:20.978626 (+ 143us) client_negotiation.cc:769] Sending connection context
0620 00:20:20.978729 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 00:20:20.978880 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":108,"threads_started":1}
I20260620 00:20:20.979346 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.969535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57618 (local address 127.0.98.60:36223)
0620 00:20:20.969761 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.969771 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.969808 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:20:20.970352 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.970361 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.970429 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.970554 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.971194 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.972247 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.973083 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.973541 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.973624 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.976209 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.976219 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.976219 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.976232 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.977626 (+ 1394us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.978168 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.978171 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.978171 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.978243 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.978459 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.978460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.978460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.978630 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.978801 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.978872 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.978952 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":96,"threads_started":1}
I20260620 00:20:20.979959 13923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:20:20.980931 13773 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } attempt: 2
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:20.988273 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.980419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46460 (local address 127.0.98.61:40813)
0620 00:20:20.980463 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.980465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.980477 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:20:20.980503 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.980504 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.980533 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.980610 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.980943 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.981572 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.982504 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.982791 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.982830 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.985169 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.985171 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.985171 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.985180 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.986878 (+ 1698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.987343 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.987345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.987345 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.987405 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.987632 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.987633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.987634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.987784 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.987861 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.988046 (+ 185us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.988133 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:20:20.988629 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.980265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:40813 (local address 127.0.0.1:46460)
0620 00:20:20.980361 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.980386 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:20.980425 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:20.980614 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:20.980615 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:20.980780 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:20.980835 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.980837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.981643 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.981644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.982411 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.982413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.982901 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.982902 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.982965 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.983697 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:20.983702 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:20.985071 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:20.986980 (+ 1909us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.986981 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.986981 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.987248 (+ 267us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.987479 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.987480 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.987480 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.987543 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.987863 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:20.987864 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:20.987913 (+ 49us) client_negotiation.cc:769] Sending connection context
0620 00:20:20.988398 (+ 485us) client_negotiation.cc:240] Negotiation successful
0620 00:20:20.988512 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81}
I20260620 00:20:20.988816 13775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.981204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57634)
0620 00:20:20.981268 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.981277 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:20.981313 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:20.981754 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:20.981755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:20.981891 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:20.981920 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.981922 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.982494 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.982495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.982978 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.982979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.983348 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.983349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.983402 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.983863 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:20.983868 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:20.985473 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:20.987587 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.987588 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.987588 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.987919 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.988128 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.988129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.988129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.988209 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.988509 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:20.988510 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:20.988555 (+ 45us) client_negotiation.cc:769] Sending connection context
0620 00:20:20.988601 (+ 46us) client_negotiation.cc:240] Negotiation successful
0620 00:20:20.988700 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50,"mutex_wait_us":1}
I20260620 00:20:20.989061 13923 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:20:20.989195 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.981203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57634 (local address 127.0.98.60:36223)
0620 00:20:20.981656 (+ 453us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.981657 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.981663 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:20:20.981677 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.981677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.981685 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.981749 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.981994 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.982418 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.983083 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.983267 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.983347 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.985591 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.985593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.985593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.985600 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.987482 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.987999 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.988000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.988001 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.988052 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.988284 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.988291 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.988292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.988434 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.989000 (+ 566us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.989040 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.989117 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425}
I20260620 00:20:20.989437 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.980354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59054 (local address 127.0.98.62:39739)
0620 00:20:20.980411 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.980413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.980542 (+ 129us) server_negotiation.cc:407] Connection header received
0620 00:20:20.980569 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.980570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.980581 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.980640 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.981263 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.981772 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.982682 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.982969 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.983027 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.986172 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.986173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.986173 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.986180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.987934 (+ 1754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.988350 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.988351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.988351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.988410 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.988596 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.988597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.988597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.989029 (+ 432us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.989234 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.989276 (+ 42us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.989355 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:20:20.991365 13773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426: Bootstrap starting.
I20260620 00:20:20.992116 13773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:20.992465 13773 log.cc:824] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:20.993631 13773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426: No bootstrap required, opened a new log
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:20:20 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:20:20.995827 13773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } }
I20260620 00:20:20.995983 13773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:20.996023 13773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95db5e0151b14547b2030d52c746c426, State: Initialized, Role: FOLLOWER
I20260620 00:20:20.996184 13773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } }
I20260620 00:20:20.996531 13773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:20.997244 13927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.989912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57648)
0620 00:20:20.989960 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:20.989970 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:20.990005 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:20.990284 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:20.990285 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:20.990421 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:20.990446 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.990447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.991059 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.991060 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.991600 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:20.991602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.992446 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.992447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:20.992654 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.993065 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:20.993069 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:20.994219 (+ 1150us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:20.996177 (+ 1958us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.996178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.996178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.996396 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.996597 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:20.996598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:20.996598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:20.996655 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:20.996935 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:20.996936 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:20.996973 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 00:20:20.997036 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 00:20:20.997131 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 00:20:20.997525 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.989955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57648 (local address 127.0.98.60:36223)
0620 00:20:20.990190 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.990192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.990197 (+ 5us) server_negotiation.cc:407] Connection header received
0620 00:20:20.990212 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.990212 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.990219 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.990282 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.990518 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.990971 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.991757 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.992313 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.992608 (+ 295us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.994305 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.994306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.994306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.994313 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.996074 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.996469 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.996470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.996470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.996519 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.996724 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.996725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.996725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.996866 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.996944 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.997359 (+ 415us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.997432 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221}
I20260620 00:20:20.997691 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:20.997771 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:20:20.998337 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:20.990061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46476 (local address 127.0.98.61:40813)
0620 00:20:20.990103 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:20.990105 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:20.990113 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:20:20.990171 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:20.990172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:20.990181 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:20.990227 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:20.990868 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.991513 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.992469 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:20.992751 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:20.992799 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:20.995282 (+ 2483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:20.995284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:20.995284 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:20.995301 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:20.997035 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.997437 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.997439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.997439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.997491 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:20.997704 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:20.997705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:20.997705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:20.997822 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:20.997933 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:20.998140 (+ 207us) server_negotiation.cc:299] Negotiation successful
0620 00:20:20.998229 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 00:20:20.999235 13940 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:20:20.999286 13940 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:20:20.999501 13923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e: Bootstrap starting.
I20260620 00:20:21.000531 13923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:21.000840 13923 log.cc:824] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:21.001114 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:20:21.001561 13923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e: No bootstrap required, opened a new log
I20260620 00:20:21.000231 13705 master_runner.cc:186] Error getting master registration for 127.0.98.61:40813: OK, instance_id { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" instance_seqno: 1781914820789274 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:20:21.003409 13705 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:20:21.003592 13923 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } }
I20260620 00:20:21.003747 13923 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:21.003793 13923 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8bf2700690664a7d88d79e6613598a5e, State: Initialized, Role: FOLLOWER
I20260620 00:20:21.003916 13923 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } }
I20260620 00:20:21.004258 13923 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:21.004209 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:21.007035 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:20:21.008023 13855 master_runner.cc:186] Error getting master registration for 127.0.98.61:40813: OK, instance_id { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" instance_seqno: 1781914820789274 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:20:21.011371 13855 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260620 00:20:21.006276 13961 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:20:21.011444 13961 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:20:21.012198 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:21.002640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.0.98.60:36223)
0620 00:20:21.002676 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:21.002678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:21.003618 (+ 940us) server_negotiation.cc:407] Connection header received
0620 00:20:21.003692 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:21.003693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:21.003702 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:21.003775 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:21.004618 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.005237 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.006324 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.006614 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.006684 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:21.008867 (+ 2183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:21.008868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:21.008869 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:21.008875 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:21.010636 (+ 1761us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:21.011050 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:21.011051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:21.011052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:21.011131 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:21.011419 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:21.011419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:21.011420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:21.011579 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:21.011674 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:21.012005 (+ 331us) server_negotiation.cc:299] Negotiation successful
0620 00:20:21.012082 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:20:21.013670 13853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:21.003232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46492 (local address 127.0.98.61:40813)
0620 00:20:21.003268 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:21.003270 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:21.005989 (+ 2719us) server_negotiation.cc:407] Connection header received
0620 00:20:21.006083 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:21.006084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:21.006096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:21.006125 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:21.006838 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.007549 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.008568 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.008872 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.008917 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:21.010481 (+ 1564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:21.010483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:21.010483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:21.010493 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:21.012268 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:21.012821 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:21.012822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:21.012822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:21.012879 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:21.013083 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:21.013084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:21.013084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:21.013221 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:21.013343 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:21.013494 (+ 151us) server_negotiation.cc:299] Negotiation successful
0620 00:20:21.013580 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:20:21.018234 13776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:21.002956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59058 (local address 127.0.98.62:39739)
0620 00:20:21.002989 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:21.002991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:21.005831 (+ 2840us) server_negotiation.cc:407] Connection header received
0620 00:20:21.005902 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:21.005903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:21.005913 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:21.005968 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:21.011540 (+ 5572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.012250 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.013389 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.013663 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.013726 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:21.015267 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:21.015268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:21.015268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:21.015276 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:21.017010 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:21.017406 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:21.017408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:21.017408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:21.017468 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:21.017679 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:21.017680 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:21.017680 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:21.017815 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:21.017921 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:21.018055 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 00:20:21.018139 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 00:20:21.027380 13848 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } attempt: 3
Jun 20 00:20:21 dist-test-slave-r81b krb5kdc[13692](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:20:21.034158 13950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:21.034261 13950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } }
I20260620 00:20:21.034504 13950 leader_election.cc:288] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 95db5e0151b14547b2030d52c746c426 (127.0.98.62:39739), cd8b8e977a1f4f1086d38e150d9af01b (127.0.98.61:40813)
I20260620 00:20:21.035048 13747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bf2700690664a7d88d79e6613598a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95db5e0151b14547b2030d52c746c426" is_pre_election: true
I20260620 00:20:21.035207 13747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf2700690664a7d88d79e6613598a5e in term 0.
I20260620 00:20:21.035450 13871 leader_election.cc:302] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf2700690664a7d88d79e6613598a5e, 95db5e0151b14547b2030d52c746c426; no voters:
I20260620 00:20:21.035590 13950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:20:21.035661 13950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:20:21.035686 13950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:21.036617 13950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } }
I20260620 00:20:21.036747 13950 leader_election.cc:288] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [CANDIDATE]: Term 1 election: Requested vote from peers 95db5e0151b14547b2030d52c746c426 (127.0.98.62:39739), cd8b8e977a1f4f1086d38e150d9af01b (127.0.98.61:40813)
I20260620 00:20:21.037093 13747 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bf2700690664a7d88d79e6613598a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "95db5e0151b14547b2030d52c746c426"
I20260620 00:20:21.037192 13747 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:21.038064 13747 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf2700690664a7d88d79e6613598a5e in term 1.
I20260620 00:20:21.038262 13871 leader_election.cc:302] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8bf2700690664a7d88d79e6613598a5e, 95db5e0151b14547b2030d52c746c426; no voters:
I20260620 00:20:21.038367 13950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:20:21.038542 13950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [term 1 LEADER]: Becoming Leader. State: Replica: 8bf2700690664a7d88d79e6613598a5e, State: Running, Role: LEADER
I20260620 00:20:21.038636 13950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } }
I20260620 00:20:21.039067 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bf2700690664a7d88d79e6613598a5e. Latest consensus state: current_term: 1 leader_uuid: "8bf2700690664a7d88d79e6613598a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:21.039279 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:21.039649 13965 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:20:21.039978 13965 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:20:21.041734 13850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:21.027690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36223 (local address 127.0.0.1:57668)
0620 00:20:21.027740 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:21.027751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:21.027788 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:21.027948 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:21.027949 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:21.028161 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:21.028227 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:21.028229 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.029176 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.029177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:21.030031 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:21.030034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.030513 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.030514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:21.030577 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:21.031102 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:21.031107 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:21.032788 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:21.040350 (+ 7562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:21.040352 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:21.040352 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:21.040604 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:21.040922 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:21.040923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:21.040923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:21.040992 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:21.041396 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:21.041397 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:21.041456 (+ 59us) client_negotiation.cc:769] Sending connection context
0620 00:20:21.041503 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:20:21.041615 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260620 00:20:21.042299 13926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:21.027737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57668 (local address 127.0.98.60:36223)
0620 00:20:21.027778 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:21.027780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:21.027789 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:20:21.027842 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:21.027843 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:21.027853 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:21.027928 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:21.028316 (+ 388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.029082 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.030126 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:21.030429 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:21.030502 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:21.032912 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:21.032913 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:21.032914 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:21.032922 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:21.040225 (+ 7303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:21.040746 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:21.040747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:21.040748 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:21.040831 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:21.041070 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:21.041071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:21.041071 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:21.041300 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:21.041961 (+ 661us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:21.042027 (+ 66us) server_negotiation.cc:299] Negotiation successful
0620 00:20:21.042152 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:20:21.043854 13848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b: Bootstrap starting.
I20260620 00:20:21.044632 13848 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:21.044948 13848 log.cc:824] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:21.045562 13747 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [term 1 FOLLOWER]: Refusing update from remote peer 8bf2700690664a7d88d79e6613598a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:21.045987 13963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:21.046660 13848 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b: No bootstrap required, opened a new log
I20260620 00:20:21.048215 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bf2700690664a7d88d79e6613598a5e. Latest consensus state: current_term: 1 leader_uuid: "8bf2700690664a7d88d79e6613598a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:21.048327 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:20:21.048974 13848 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } }
I20260620 00:20:21.049104 13848 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:21.049144 13848 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd8b8e977a1f4f1086d38e150d9af01b, State: Initialized, Role: FOLLOWER
I20260620 00:20:21.049254 13848 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } }
I20260620 00:20:21.049533 13848 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:21.049723 13820 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:21.050652 13820 raft_consensus.cc:1261] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [term 1 FOLLOWER]: Refusing update from remote peer 8bf2700690664a7d88d79e6613598a5e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:21.050845 13822 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bf2700690664a7d88d79e6613598a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" is_pre_election: true
I20260620 00:20:21.050961 13822 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8bf2700690664a7d88d79e6613598a5e in term 1.
I20260620 00:20:21.050935 13963 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [LEADER]: Connected to new peer: Peer: permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:21.051162 13821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8bf2700690664a7d88d79e6613598a5e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cd8b8e977a1f4f1086d38e150d9af01b"
I20260620 00:20:21.052750 13821 raft_consensus.cc:2454] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8bf2700690664a7d88d79e6613598a5e in term 1.
I20260620 00:20:21.054077 13970 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8bf2700690664a7d88d79e6613598a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:21.054183 13970 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:20:21.054561 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8bf2700690664a7d88d79e6613598a5e. Latest consensus state: current_term: 1 leader_uuid: "8bf2700690664a7d88d79e6613598a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:21.054658 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:20:21.055199 13981 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:20:21.055255 13981 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:20:21.055826 13963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bf2700690664a7d88d79e6613598a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:21.055912 13963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:21.055999 13929 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bf2700690664a7d88d79e6613598a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:21.056102 13929 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:20:21.056322 13966 mvcc.cc:205] Tried to move back new op lower bound from 7298723106997854208 to 7298723106974887936. Current Snapshot: MvccSnapshot[applied={T|T < 7298723106997854208}]
I20260620 00:20:21.056877 13968 mvcc.cc:205] Tried to move back new op lower bound from 7298723106997854208 to 7298723106974887936. Current Snapshot: MvccSnapshot[applied={T|T < 7298723106997854208}]
I20260620 00:20:21.064745 13965 catalog_manager.cc:1347] Generated new cluster ID: 118d5fa4806d422d9642b9f4a0b9c567
I20260620 00:20:21.064828 13965 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:20:21.065326 13950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8bf2700690664a7d88d79e6613598a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:21.065433 13950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:21.066094 13983 mvcc.cc:205] Tried to move back new op lower bound from 7298723106997854208 to 7298723106974887936. Current Snapshot: MvccSnapshot[applied={T|T < 7298723106997854208}]
I20260620 00:20:21.066185 13982 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8bf2700690664a7d88d79e6613598a5e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "95db5e0151b14547b2030d52c746c426" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39739 } } peers { permanent_uuid: "cd8b8e977a1f4f1086d38e150d9af01b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 40813 } } peers { permanent_uuid: "8bf2700690664a7d88d79e6613598a5e" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36223 } } }
I20260620 00:20:21.066254 13982 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:20:21.075469 13965 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:20:21.076257 13965 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:20:21.090415 13965 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8bf2700690664a7d88d79e6613598a5e: Generated new TSK 0
I20260620 00:20:21.090674 13965 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:20:21.999765 13940 catalog_manager.cc:1259] Loaded cluster ID: 118d5fa4806d422d9642b9f4a0b9c567
I20260620 00:20:21.999846 13940 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426: loading cluster ID for follower catalog manager: success
I20260620 00:20:22.000801 13940 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426: acquiring CA information for follower catalog manager: success
I20260620 00:20:22.001143 13940 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 95db5e0151b14547b2030d52c746c426: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:20:22.055717 13981 catalog_manager.cc:1259] Loaded cluster ID: 118d5fa4806d422d9642b9f4a0b9c567
I20260620 00:20:22.055831 13981 catalog_manager.cc:1552] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b: loading cluster ID for follower catalog manager: success
I20260620 00:20:22.056727 13981 catalog_manager.cc:1574] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b: acquiring CA information for follower catalog manager: success
I20260620 00:20:22.057049 13981 catalog_manager.cc:1602] T 00000000000000000000000000000000 P cd8b8e977a1f4f1086d38e150d9af01b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:20:22.144939 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 13705
I20260620 00:20:22.150282 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 13780
I20260620 00:20:22.155268 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 13855
2026-06-20T00:20:22Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2813 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:20:22 dist-test-slave-r81b krb5kdc[13995](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:20:22 dist-test-slave-r81b krb5kdc[13995](info): set up 2 sockets
Jun 20 00:20:22 dist-test-slave-r81b krb5kdc[13995](info): commencing operation
krb5kdc: starting...
W20260620 00:20:23.204335 392 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:20:23 dist-test-slave-r81b krb5kdc[13995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:20:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:20:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:20:23.322969 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41955
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40041
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:23.404794 14008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:23.405066 14008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:23.405107 14008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:23.406314 14008 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:23.406391 14008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:23.406416 14008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:23.406440 14008 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:23.406467 14008 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:23.407992 14008 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40041
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:23.408314 14008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:23.408551 14008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:23.413692 14013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:23.413677 14016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:23.414032 14008 server_base.cc:1034] running on GCE node
W20260620 00:20:23.413677 14014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:23.414381 14008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:23.414721 14008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:23.415940 14008 hybrid_clock.cc:648] HybridClock initialized: now 1781914823415917 us; error 43 us; skew 500 ppm
Jun 20 00:20:23 dist-test-slave-r81b krb5kdc[13995](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:23.418934 14008 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:20:23.419471 14008 webserver.cc:460] Webserver started at http://127.0.98.62:35865/ using document root <none> and password file <none>
I20260620 00:20:23.419718 14008 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:23.419787 14008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:23.419932 14008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:23.421072 14008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "e40f7aa0bd50421eb7c1da29e1dd8491"
format_stamp: "Formatted at 2026-06-20 00:20:23 on dist-test-slave-r81b"
server_key: "71e3e1073e79fb3057158b50fe00d88b"
server_key_iv: "609c534610bbfa17e4041ab113d94846"
server_key_version: "encryptionkey@0"
I20260620 00:20:23.421437 14008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "e40f7aa0bd50421eb7c1da29e1dd8491"
format_stamp: "Formatted at 2026-06-20 00:20:23 on dist-test-slave-r81b"
server_key: "71e3e1073e79fb3057158b50fe00d88b"
server_key_iv: "609c534610bbfa17e4041ab113d94846"
server_key_version: "encryptionkey@0"
I20260620 00:20:23.423493 14008 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260620 00:20:23.424371 14023 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:23.424595 14008 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:23.424721 14008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "e40f7aa0bd50421eb7c1da29e1dd8491"
format_stamp: "Formatted at 2026-06-20 00:20:23 on dist-test-slave-r81b"
server_key: "71e3e1073e79fb3057158b50fe00d88b"
server_key_iv: "609c534610bbfa17e4041ab113d94846"
server_key_version: "encryptionkey@0"
I20260620 00:20:23.424810 14008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:23.447395 14008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:23.449709 14008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:23.449918 14008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:23.454084 14008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41955
I20260620 00:20:23.454116 14075 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41955 every 8 connection(s)
I20260620 00:20:23.454489 14008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:20:23.455395 14076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:23.457752 14076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491: Bootstrap starting.
I20260620 00:20:23.458014 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 14008
I20260620 00:20:23.458137 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:20:23.458341 392 external_mini_cluster.cc:1428] Setting key 5bc9cb2d1453d11a7d3fa17ad42af2a1
I20260620 00:20:23.458527 14076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:23.458861 14076 log.cc:824] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:23.459514 14076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491: No bootstrap required, opened a new log
I20260620 00:20:23.461390 14076 raft_consensus.cc:348] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f7aa0bd50421eb7c1da29e1dd8491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } }
I20260620 00:20:23.461548 14076 raft_consensus.cc:374] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:23.461596 14076 raft_consensus.cc:729] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40f7aa0bd50421eb7c1da29e1dd8491, State: Initialized, Role: FOLLOWER
I20260620 00:20:23.461740 14076 consensus_queue.cc:260] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f7aa0bd50421eb7c1da29e1dd8491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } }
I20260620 00:20:23.461803 14076 raft_consensus.cc:388] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:20:23.461824 14076 raft_consensus.cc:482] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:20:23.461890 14076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:23.462441 14076 raft_consensus.cc:504] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f7aa0bd50421eb7c1da29e1dd8491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } }
I20260620 00:20:23.462561 14076 leader_election.cc:302] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e40f7aa0bd50421eb7c1da29e1dd8491; no voters:
I20260620 00:20:23.462761 14076 leader_election.cc:288] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:20:23.462783 14081 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:20:23.462903 14081 raft_consensus.cc:686] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [term 1 LEADER]: Becoming Leader. State: Replica: e40f7aa0bd50421eb7c1da29e1dd8491, State: Running, Role: LEADER
I20260620 00:20:23.462991 14081 consensus_queue.cc:237] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f7aa0bd50421eb7c1da29e1dd8491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } }
I20260620 00:20:23.463024 14076 sys_catalog.cc:564] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:23.463286 14081 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e40f7aa0bd50421eb7c1da29e1dd8491. Latest consensus state: current_term: 1 leader_uuid: "e40f7aa0bd50421eb7c1da29e1dd8491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f7aa0bd50421eb7c1da29e1dd8491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } } }
I20260620 00:20:23.463367 14081 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:23.463296 14083 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e40f7aa0bd50421eb7c1da29e1dd8491" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40f7aa0bd50421eb7c1da29e1dd8491" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } } }
I20260620 00:20:23.463523 14083 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491 [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:23.463616 14085 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:20:23.464184 14085 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:20:23 dist-test-slave-r81b krb5kdc[13995](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:23.466845 14085 catalog_manager.cc:1347] Generated new cluster ID: f6f629b9613b4e4d8bae2c481f1c6d6b
I20260620 00:20:23.466945 14085 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:20:23.469871 14079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:23.458877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48672 (local address 127.0.98.62:41955)
0620 00:20:23.459067 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:23.459078 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:23.459116 (+ 38us) server_negotiation.cc:407] Connection header received
0620 00:20:23.459615 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:23.459624 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:23.459706 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:23.459849 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:23.460580 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:23.461463 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:23.462416 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:23.463030 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:23.463432 (+ 402us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:23.465881 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:23.465886 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:23.465886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:23.465900 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:23.467878 (+ 1978us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:23.468269 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:23.468273 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:23.468273 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:23.468364 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:23.468534 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:23.468535 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:23.468535 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:23.468812 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:23.468940 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:23.469390 (+ 450us) server_negotiation.cc:299] Negotiation successful
0620 00:20:23.469517 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":78,"threads_started":1}
I20260620 00:20:23.470850 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:20:23.474299 14085 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:20:23.474785 14085 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:20:23.480443 14085 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e40f7aa0bd50421eb7c1da29e1dd8491: Generated new TSK 0
I20260620 00:20:23.480643 14085 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:20:23.525348 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 14008
2026-06-20T00:20:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1368 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:20:23 dist-test-slave-r81b krb5kdc[14103](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:20:23 dist-test-slave-r81b krb5kdc[14103](info): set up 2 sockets
Jun 20 00:20:23 dist-test-slave-r81b krb5kdc[14103](info): commencing operation
krb5kdc: starting...
W20260620 00:20:24.584455 392 mini_kdc.cc:121] Time spent starting KDC: real 1.041s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:20:24 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:20:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:20:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:20:24.703681 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33365
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:24.787254 14116 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:24.787480 14116 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:24.787504 14116 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:24.788883 14116 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:24.788935 14116 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:24.788950 14116 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:24.788964 14116 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:24.788976 14116 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:24.790530 14116 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14116
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:24.790807 14116 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:24.791044 14116 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:20:24.796236 14116 server_base.cc:1034] running on GCE node
W20260620 00:20:24.796221 14121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:24.796226 14124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:24.796394 14122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:24.796717 14116 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:24.797035 14116 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:24.798203 14116 hybrid_clock.cc:648] HybridClock initialized: now 1781914824798186 us; error 37 us; skew 500 ppm
Jun 20 00:20:24 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:24.801012 14116 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:20:24.801594 14116 webserver.cc:460] Webserver started at http://127.0.98.62:42431/ using document root <none> and password file <none>
I20260620 00:20:24.801864 14116 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:24.801939 14116 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:24.802103 14116 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:24.803249 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "178515fe049d475e989183850879c65e"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "35ccdfe1795e786876b6c32b7301dd91"
server_key_iv: "a370ca26dccbe09c6820dd5684979ee9"
server_key_version: "encryptionkey@0"
I20260620 00:20:24.803682 14116 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "178515fe049d475e989183850879c65e"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "35ccdfe1795e786876b6c32b7301dd91"
server_key_iv: "a370ca26dccbe09c6820dd5684979ee9"
server_key_version: "encryptionkey@0"
I20260620 00:20:24.805292 14116 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.002s
I20260620 00:20:24.806092 14131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:24.806422 14116 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:24.806536 14116 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "178515fe049d475e989183850879c65e"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "35ccdfe1795e786876b6c32b7301dd91"
server_key_iv: "a370ca26dccbe09c6820dd5684979ee9"
server_key_version: "encryptionkey@0"
I20260620 00:20:24.806617 14116 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:24.814810 14116 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:24.817226 14116 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:24.817394 14116 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:24.821732 14116 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33365
I20260620 00:20:24.821767 14183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33365 every 8 connection(s)
I20260620 00:20:24.822146 14116 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:20:24.823108 14184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:24.825536 14184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Bootstrap starting.
I20260620 00:20:24.826119 14184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:24.826416 14184 log.cc:824] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:24.827203 14184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: No bootstrap required, opened a new log
I20260620 00:20:24.828812 14184 raft_consensus.cc:348] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:24.828902 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 14116
I20260620 00:20:24.828979 14184 raft_consensus.cc:374] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:24.829028 14184 raft_consensus.cc:729] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178515fe049d475e989183850879c65e, State: Initialized, Role: FOLLOWER
I20260620 00:20:24.829062 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:20:24.829231 14184 consensus_queue.cc:260] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:24.829303 392 external_mini_cluster.cc:1428] Setting key 1fe6f5cb537452425c9ce901592bf7bb
I20260620 00:20:24.829334 14184 raft_consensus.cc:388] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:20:24.829380 14184 raft_consensus.cc:482] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:20:24.829430 14184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:24.830152 14184 raft_consensus.cc:504] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:24.830273 14184 leader_election.cc:302] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 178515fe049d475e989183850879c65e; no voters:
I20260620 00:20:24.830441 14184 leader_election.cc:288] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:20:24.830469 14189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:20:24.830628 14189 raft_consensus.cc:686] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 1 LEADER]: Becoming Leader. State: Replica: 178515fe049d475e989183850879c65e, State: Running, Role: LEADER
I20260620 00:20:24.830687 14184 sys_catalog.cc:564] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:24.830775 14189 consensus_queue.cc:237] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:24.831200 14190 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "178515fe049d475e989183850879c65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } } }
I20260620 00:20:24.831192 14191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 178515fe049d475e989183850879c65e. Latest consensus state: current_term: 1 leader_uuid: "178515fe049d475e989183850879c65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } } }
I20260620 00:20:24.831287 14190 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:24.831291 14191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:24.831597 14197 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:20:24.832620 14197 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:20:24.834287 14197 catalog_manager.cc:1347] Generated new cluster ID: 987fd1b0ac5042cab0f89b3bdddd68a6
I20260620 00:20:24.834342 14197 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:20:24 dist-test-slave-r81b krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:24.839970 14197 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:20:24.840713 14197 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:20:24.841285 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:24.830185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41650 (local address 127.0.98.62:33365)
0620 00:20:24.830397 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:24.830408 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:24.830443 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:20:24.830569 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:24.830577 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:24.830643 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:24.831651 (+ 1008us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:24.831767 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:24.833515 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:24.834568 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:24.835012 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:24.835155 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:24.837716 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:24.837720 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:24.837720 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:24.837734 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:24.839510 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:24.839948 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:24.839951 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:24.839952 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:24.840065 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:24.840307 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:24.840308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:24.840309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:24.840575 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:24.840778 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:24.840855 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:20:24.841116 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":157,"thread_start_us":106,"threads_started":1}
I20260620 00:20:24.848868 14197 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Generated new TSK 0
I20260620 00:20:24.849088 14197 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:20:24.879637 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33365
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:24.969448 14210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:24.969669 14210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:24.969695 14210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:24.970886 14210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:24.970930 14210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:24.970959 14210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:20:24.972534 14210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:24.972792 14210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:24.973039 14210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:24.978219 14216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:24.978174 14215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:24.978189 14218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:24.978871 14210 server_base.cc:1034] running on GCE node
I20260620 00:20:24.979071 14210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:24.979331 14210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:24.980479 14210 hybrid_clock.cc:648] HybridClock initialized: now 1781914824980461 us; error 32 us; skew 500 ppm
Jun 20 00:20:24 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:24.983562 14210 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:20:24.984088 14210 webserver.cc:460] Webserver started at http://127.0.98.1:39663/ using document root <none> and password file <none>
I20260620 00:20:24.984375 14210 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:24.984431 14210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:24.984603 14210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:24.985960 14210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "edbcb73a595a2ecef8056dfe349f3002"
server_key_iv: "74df5ddcc5f6a748a3d3a6c915eb4499"
server_key_version: "encryptionkey@0"
I20260620 00:20:24.986379 14210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "edbcb73a595a2ecef8056dfe349f3002"
server_key_iv: "74df5ddcc5f6a748a3d3a6c915eb4499"
server_key_version: "encryptionkey@0"
I20260620 00:20:24.987934 14210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:20:24.988797 14225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:24.988973 14210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:24.989060 14210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "edbcb73a595a2ecef8056dfe349f3002"
server_key_iv: "74df5ddcc5f6a748a3d3a6c915eb4499"
server_key_version: "encryptionkey@0"
I20260620 00:20:24.989117 14210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:25.004880 14210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:25.007302 14210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:25.007432 14210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:25.007648 14210 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:25.007993 14210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:25.008036 14210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.008060 14210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:25.008111 14210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.014046 14210 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45347
I20260620 00:20:25.014135 14338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45347 every 8 connection(s)
I20260620 00:20:25.014467 14210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:20:25.015458 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 14210
I20260620 00:20:25.015585 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:20:25.015830 392 external_mini_cluster.cc:1428] Setting key c7969d10737004e4d22f47d41eb51a28
Jun 20 00:20:25 dist-test-slave-r81b krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914824, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:20:25.026608 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.015390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33961 (local address 127.0.98.62:33365)
0620 00:20:25.015440 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.015442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.015600 (+ 158us) server_negotiation.cc:407] Connection header received
0620 00:20:25.016373 (+ 773us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.016374 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.016386 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.016445 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:25.017888 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.018728 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.019829 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.020200 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.020287 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.022965 (+ 2678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:25.022967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:25.022967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:25.022977 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:25.024920 (+ 1943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.025457 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.025459 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.025459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.025531 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.025805 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.025806 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.025807 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.025932 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:25.025999 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.026451 (+ 452us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.026538 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:20:25.026784 14341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.015246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:33961)
0620 00:20:25.015445 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.015497 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.016256 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.016535 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.016540 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.017479 (+ 939us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:25.017761 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.017767 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.018844 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.018845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.019721 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.019724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.020291 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.020292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.020436 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.021047 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:25.021056 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:25.022858 (+ 1802us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:25.025009 (+ 2151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:25.025016 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:25.025018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:25.025374 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:25.025612 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:25.025612 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:25.025613 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:25.025718 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:25.026038 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:25.026045 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:25.026251 (+ 206us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.026410 (+ 159us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.026550 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":108,"threads_started":1}
I20260620 00:20:25.027486 14339 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33365
I20260620 00:20:25.027596 14339 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:25.027851 14339 heartbeater.cc:511] Master 127.0.98.62:33365 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:20:25.028540 14148 ts_manager.cc:194] Registered new tserver with Master: c1d8a755cb574d87a8a0eebd7e3e3dee (127.0.98.1:45347)
I20260620 00:20:25.029217 14148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33961
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:20:25.043586 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33365
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:25.126606 14344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:25.126835 14344 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:25.126860 14344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:25.128108 14344 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:25.128178 14344 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:25.128207 14344 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:20:25.129684 14344 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14344
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:25.129951 14344 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:25.130189 14344 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:25.135813 14349 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:25.135924 14344 server_base.cc:1034] running on GCE node
W20260620 00:20:25.135833 14352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:25.135839 14350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:25.136324 14344 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:25.136583 14344 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:25.137779 14344 hybrid_clock.cc:648] HybridClock initialized: now 1781914825137756 us; error 53 us; skew 500 ppm
Jun 20 00:20:25 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:25.140552 14344 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:20:25.141047 14344 webserver.cc:460] Webserver started at http://127.0.98.2:41959/ using document root <none> and password file <none>
I20260620 00:20:25.141312 14344 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:25.141386 14344 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:25.141537 14344 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:25.142623 14344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "14b36fcf88134e9f801650fa8758fc2b"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "6eee1e52848b85446941c0c06fadab94"
server_key_iv: "90706db93e8bf8c18d332e021dd1a60a"
server_key_version: "encryptionkey@0"
I20260620 00:20:25.143002 14344 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "14b36fcf88134e9f801650fa8758fc2b"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "6eee1e52848b85446941c0c06fadab94"
server_key_iv: "90706db93e8bf8c18d332e021dd1a60a"
server_key_version: "encryptionkey@0"
I20260620 00:20:25.144390 14344 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:20:25.145139 14359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.145314 14344 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:20:25.145421 14344 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "14b36fcf88134e9f801650fa8758fc2b"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "6eee1e52848b85446941c0c06fadab94"
server_key_iv: "90706db93e8bf8c18d332e021dd1a60a"
server_key_version: "encryptionkey@0"
I20260620 00:20:25.145494 14344 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:25.158553 14344 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:25.160727 14344 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:25.160892 14344 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:25.161135 14344 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:25.161455 14344 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:25.161511 14344 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.161556 14344 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:25.161578 14344 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.167409 14344 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34485
I20260620 00:20:25.167448 14472 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34485 every 8 connection(s)
I20260620 00:20:25.167819 14344 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:20:25.169034 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 14344
I20260620 00:20:25.169119 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:20:25.169329 392 external_mini_cluster.cc:1428] Setting key 44c43478aea1af6e436beaea458781be
Jun 20 00:20:25 dist-test-slave-r81b krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:20:25.178416 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.168625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57249 (local address 127.0.98.62:33365)
0620 00:20:25.168713 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.168715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.168894 (+ 179us) server_negotiation.cc:407] Connection header received
0620 00:20:25.169506 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.169508 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.169520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.169572 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:25.170746 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.171542 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.172673 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.172905 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.172989 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.175233 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:25.175234 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:25.175235 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:25.175243 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:25.176748 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.177173 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.177174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.177175 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.177257 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.177631 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.177632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.177632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.177781 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:25.177873 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.178212 (+ 339us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.178310 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260620 00:20:25.178643 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.168579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:57249)
0620 00:20:25.168757 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.168796 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.169402 (+ 606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.169658 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.169663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.170354 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:25.170618 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.170623 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.171632 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.171633 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.172572 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.172575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.172997 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.172998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.173139 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.173711 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:25.173719 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:25.175096 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:25.176849 (+ 1753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:25.176856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:25.176859 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:25.177091 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:25.177338 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:25.177339 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:25.177340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:25.177504 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:25.177875 (+ 371us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:25.177880 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:25.178005 (+ 125us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.178187 (+ 182us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.178399 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260620 00:20:25.179220 14473 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33365
I20260620 00:20:25.179349 14473 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 00:20:25.179559 14473 heartbeater.cc:511] Master 127.0.98.62:33365 requested a full tablet report, sending...
I20260620 00:20:25.180089 14148 ts_manager.cc:194] Registered new tserver with Master: 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485)
I20260620 00:20:25.180534 14148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57249
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:20:25.195147 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33365
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:25.278261 14478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:25.278498 14478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:25.278524 14478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:25.279657 14478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:25.279696 14478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:25.279723 14478 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:20:25.281255 14478 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:25.281459 14478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:25.281653 14478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:25.287015 14484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:25.287043 14478 server_base.cc:1034] running on GCE node
W20260620 00:20:25.287020 14486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:25.287015 14483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:25.287437 14478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:25.287688 14478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:25.288864 14478 hybrid_clock.cc:648] HybridClock initialized: now 1781914825288851 us; error 30 us; skew 500 ppm
Jun 20 00:20:25 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:25.291499 14478 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:20:25.291944 14478 webserver.cc:460] Webserver started at http://127.0.98.3:38059/ using document root <none> and password file <none>
I20260620 00:20:25.292236 14478 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:25.292284 14478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:25.292431 14478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:25.293493 14478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "aec66acaa4b5442f8226c144a10d3a8f"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "85787f21c884e2fc88b05d3bdbe0df95"
server_key_iv: "c09f8c17d5301810c4af2a14e43e8dfd"
server_key_version: "encryptionkey@0"
I20260620 00:20:25.293851 14478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "aec66acaa4b5442f8226c144a10d3a8f"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "85787f21c884e2fc88b05d3bdbe0df95"
server_key_iv: "c09f8c17d5301810c4af2a14e43e8dfd"
server_key_version: "encryptionkey@0"
I20260620 00:20:25.295365 14478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:20:25.296289 14493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.296486 14478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:25.296592 14478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "aec66acaa4b5442f8226c144a10d3a8f"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "85787f21c884e2fc88b05d3bdbe0df95"
server_key_iv: "c09f8c17d5301810c4af2a14e43e8dfd"
server_key_version: "encryptionkey@0"
I20260620 00:20:25.296679 14478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:25.318065 14478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:25.320042 14478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:25.320286 14478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:25.320494 14478 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:25.320817 14478 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:25.320876 14478 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.320922 14478 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:25.320945 14478 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.326730 14478 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39915
I20260620 00:20:25.326776 14606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39915 every 8 connection(s)
I20260620 00:20:25.327095 14478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:20:25.330404 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 14478
I20260620 00:20:25.330547 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:20:25.330772 392 external_mini_cluster.cc:1428] Setting key af52550be2aec8d6a29a7711f1caf5bf
Jun 20 00:20:25 dist-test-slave-r81b krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:25.337543 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.327835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33197 (local address 127.0.98.62:33365)
0620 00:20:25.327920 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.327923 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.328081 (+ 158us) server_negotiation.cc:407] Connection header received
0620 00:20:25.328653 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.328655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.328667 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.328715 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:25.329691 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.330239 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.331642 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.331947 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.332037 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.334373 (+ 2336us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:25.334376 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:25.334376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:25.334385 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:25.335868 (+ 1483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.336437 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.336439 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.336439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.336506 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.336794 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.336795 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.336795 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.336918 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:25.336991 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.337325 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.337436 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260620 00:20:25.337786 14609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.327790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:33197)
0620 00:20:25.327950 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.327987 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.328545 (+ 558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.328793 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.328796 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.329387 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:25.329583 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.329589 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.330339 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.330341 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.331564 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.331567 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.332048 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.332049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.332227 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.332747 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:25.332756 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:25.334268 (+ 1512us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:25.335994 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:25.336002 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:25.336005 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:25.336353 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:25.336575 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:25.336576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:25.336576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:25.336699 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:25.336991 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:25.337035 (+ 44us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:25.337180 (+ 145us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.337288 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.337511 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
I20260620 00:20:25.338312 14607 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33365
I20260620 00:20:25.338423 14607 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:25.338611 14607 heartbeater.cc:511] Master 127.0.98.62:33365 requested a full tablet report, sending...
I20260620 00:20:25.339044 14148 ts_manager.cc:194] Registered new tserver with Master: aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915)
I20260620 00:20:25.339457 14148 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33197
I20260620 00:20:25.343096 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:20:25.350786 14188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.344565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.98.62:33365)
0620 00:20:25.344621 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.344624 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.344770 (+ 146us) server_negotiation.cc:407] Connection header received
0620 00:20:25.344811 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.344811 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.344820 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.344871 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:25.345378 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.345856 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.346623 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.346822 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.346886 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.348014 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:25.348015 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:25.348015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:25.348022 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:25.349683 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.350007 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.350007 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.350008 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.350072 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.350317 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.350317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.350318 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.350453 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:25.350530 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.350633 (+ 103us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.350689 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:20:25.352406 14148 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:20:25.353968 14148 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:20:25.360882 14621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.356422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34485 (local address 127.0.0.1:48120)
0620 00:20:25.356813 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.356823 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.356876 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.357123 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.357126 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.357129 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:25.357207 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.357210 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.358649 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.358650 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.359679 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.359681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.360615 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.360617 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.360683 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.360687 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.360743 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.360788 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":3,"thread_start_us":67,"threads_started":1}
I20260620 00:20:25.360883 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.356536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48120 (local address 127.0.98.2:34485)
0620 00:20:25.356733 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.356739 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.356889 (+ 150us) server_negotiation.cc:407] Connection header received
0620 00:20:25.356945 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.356948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.356966 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.357305 (+ 339us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:25.357388 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.358533 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.359762 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.360520 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.360616 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.360675 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.360757 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.360798 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":79,"threads_started":1}
I20260620 00:20:25.360980 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.356841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43912 (local address 127.0.98.3:39915)
0620 00:20:25.357040 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.357045 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.357056 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:20:25.357095 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.357097 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.357113 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.357186 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:25.357404 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.358268 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.359763 (+ 1495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.360376 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.360449 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.360519 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.360840 (+ 321us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.360888 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":117,"threads_started":1}
I20260620 00:20:25.361383 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.357192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41136 (local address 127.0.98.1:45347)
0620 00:20:25.357364 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.357369 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.357632 (+ 263us) server_negotiation.cc:407] Connection header received
0620 00:20:25.357685 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.357687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.357704 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.357788 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:25.357919 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.358884 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.360223 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.360960 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.361032 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.361091 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.361280 (+ 189us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.361320 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":94,"threads_started":1}
I20260620 00:20:25.361346 14618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.356682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45347 (local address 127.0.0.1:41136)
0620 00:20:25.357449 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.357461 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.357518 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.357804 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.357805 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.357808 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:25.357856 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.357857 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.359024 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.359025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.360098 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.360101 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.361130 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.361131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.361186 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.361190 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.361244 (+ 54us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.361281 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":79,"threads_started":1}
I20260620 00:20:25.361820 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.356422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39915 (local address 127.0.0.1:43912)
0620 00:20:25.356582 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.356628 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.356705 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.357263 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.357265 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.357267 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:25.357323 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.357324 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.358363 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.358364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.359657 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.359661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.360465 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.360466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.360564 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.360585 (+ 21us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.360663 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.360726 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":99,"threads_started":1}
I20260620 00:20:25.362358 14407 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5590dc6ca04fe3b52e58e5eeec819b (DEFAULT_TABLE table=test-table [id=3ed704e2227740aabd186bcc84949a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:25.362547 14541 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5590dc6ca04fe3b52e58e5eeec819b (DEFAULT_TABLE table=test-table [id=3ed704e2227740aabd186bcc84949a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:25.362798 14407 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5590dc6ca04fe3b52e58e5eeec819b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:25.362921 14541 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5590dc6ca04fe3b52e58e5eeec819b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:25.365759 14624 tablet_bootstrap.cc:492] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: Bootstrap starting.
I20260620 00:20:25.366508 14624 tablet_bootstrap.cc:654] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:25.366379 14273 tablet_service.cc:1467] Processing CreateTablet for tablet 4a5590dc6ca04fe3b52e58e5eeec819b (DEFAULT_TABLE table=test-table [id=3ed704e2227740aabd186bcc84949a6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:25.366758 14273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4a5590dc6ca04fe3b52e58e5eeec819b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:25.367161 14624 log.cc:824] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:25.367969 14624 tablet_bootstrap.cc:492] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: No bootstrap required, opened a new log
I20260620 00:20:25.368243 14624 ts_tablet_manager.cc:1397] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:20:25.370445 14627 tablet_bootstrap.cc:492] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Bootstrap starting.
I20260620 00:20:25.370587 14625 tablet_bootstrap.cc:492] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Bootstrap starting.
I20260620 00:20:25.370486 14624 raft_consensus.cc:348] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.370684 14624 raft_consensus.cc:374] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:25.370760 14624 raft_consensus.cc:729] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec66acaa4b5442f8226c144a10d3a8f, State: Initialized, Role: FOLLOWER
I20260620 00:20:25.370924 14624 consensus_queue.cc:260] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.371233 14624 ts_tablet_manager.cc:1428] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:20:25.371244 14627 tablet_bootstrap.cc:654] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:25.371320 14625 tablet_bootstrap.cc:654] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:25.371376 14607 heartbeater.cc:503] Master 127.0.98.62:33365 was elected leader, sending a full tablet report...
I20260620 00:20:25.371794 14627 log.cc:824] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:25.371907 14625 log.cc:824] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:25.372601 14627 tablet_bootstrap.cc:492] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: No bootstrap required, opened a new log
I20260620 00:20:25.372673 14627 ts_tablet_manager.cc:1397] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:20:25.372845 14625 tablet_bootstrap.cc:492] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: No bootstrap required, opened a new log
I20260620 00:20:25.372905 14625 ts_tablet_manager.cc:1397] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:20:25.374074 14627 raft_consensus.cc:348] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.374222 14627 raft_consensus.cc:374] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:25.374248 14627 raft_consensus.cc:729] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d8a755cb574d87a8a0eebd7e3e3dee, State: Initialized, Role: FOLLOWER
I20260620 00:20:25.374341 14627 consensus_queue.cc:260] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.374239 14625 raft_consensus.cc:348] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.374420 14625 raft_consensus.cc:374] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:25.374459 14625 raft_consensus.cc:729] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b36fcf88134e9f801650fa8758fc2b, State: Initialized, Role: FOLLOWER
I20260620 00:20:25.374558 14627 ts_tablet_manager.cc:1428] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:20:25.374599 14625 consensus_queue.cc:260] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.374665 14339 heartbeater.cc:503] Master 127.0.98.62:33365 was elected leader, sending a full tablet report...
I20260620 00:20:25.374922 14625 ts_tablet_manager.cc:1428] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:20:25.375039 14473 heartbeater.cc:503] Master 127.0.98.62:33365 was elected leader, sending a full tablet report...
W20260620 00:20:25.418431 14474 tablet.cc:2369] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:20:25.508301 14632 raft_consensus.cc:482] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:25.508471 14632 raft_consensus.cc:504] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.508936 14632 leader_election.cc:288] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915), c1d8a755cb574d87a8a0eebd7e3e3dee (127.0.98.1:45347)
I20260620 00:20:25.513336 14623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.509433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48939 (local address 127.0.98.1:45347)
0620 00:20:25.509497 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.509499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.509638 (+ 139us) server_negotiation.cc:407] Connection header received
0620 00:20:25.509719 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.509720 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.509733 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.509825 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:25.509993 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.510650 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.512291 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.512874 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.512948 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.512960 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.513222 (+ 262us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.513262 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:20:25.513366 14622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.509257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45577 (local address 127.0.98.3:39915)
0620 00:20:25.509382 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.509384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.509395 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:20:25.509430 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.509431 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.509440 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.509527 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:25.509737 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.510574 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.512308 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.512867 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.512948 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.512960 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.513221 (+ 261us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.513262 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260620 00:20:25.513352 14475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.509161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39915 (local address 127.0.98.2:45577)
0620 00:20:25.509229 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.509242 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.509295 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.509555 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.509556 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.509559 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:25.509663 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.509665 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.510694 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.510695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.512121 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.512126 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.512980 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.512981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.513077 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.513079 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.513197 (+ 118us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.513247 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":65}
I20260620 00:20:25.513360 14633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.509337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45347 (local address 127.0.98.2:48939)
0620 00:20:25.509569 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.509579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.509641 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.509830 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.509830 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.509845 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:25.509895 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.509896 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.510710 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.510710 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.512200 (+ 1490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.512202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.512980 (+ 778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.512981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.513072 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.513078 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.513201 (+ 123us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.513247 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":89,"thread_start_us":119,"threads_started":1}
I20260620 00:20:25.513876 14561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5590dc6ca04fe3b52e58e5eeec819b" candidate_uuid: "14b36fcf88134e9f801650fa8758fc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec66acaa4b5442f8226c144a10d3a8f" is_pre_election: true
I20260620 00:20:25.513877 14293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5590dc6ca04fe3b52e58e5eeec819b" candidate_uuid: "14b36fcf88134e9f801650fa8758fc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" is_pre_election: true
I20260620 00:20:25.514050 14561 raft_consensus.cc:2454] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b36fcf88134e9f801650fa8758fc2b in term 0.
I20260620 00:20:25.514050 14293 raft_consensus.cc:2454] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b36fcf88134e9f801650fa8758fc2b in term 0.
I20260620 00:20:25.514302 14360 leader_election.cc:302] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b36fcf88134e9f801650fa8758fc2b, aec66acaa4b5442f8226c144a10d3a8f; no voters:
I20260620 00:20:25.514447 14632 raft_consensus.cc:2781] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:20:25.514546 14632 raft_consensus.cc:482] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:20:25.514586 14632 raft_consensus.cc:3037] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Advancing to term 1
W20260620 00:20:25.515173 14340 tablet.cc:2369] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:20:25.515424 14632 raft_consensus.cc:504] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.515576 14632 leader_election.cc:288] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [CANDIDATE]: Term 1 election: Requested vote from peers aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915), c1d8a755cb574d87a8a0eebd7e3e3dee (127.0.98.1:45347)
I20260620 00:20:25.515802 14561 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5590dc6ca04fe3b52e58e5eeec819b" candidate_uuid: "14b36fcf88134e9f801650fa8758fc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec66acaa4b5442f8226c144a10d3a8f"
I20260620 00:20:25.515790 14293 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4a5590dc6ca04fe3b52e58e5eeec819b" candidate_uuid: "14b36fcf88134e9f801650fa8758fc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee"
I20260620 00:20:25.515897 14561 raft_consensus.cc:3037] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:25.515893 14293 raft_consensus.cc:3037] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:25.516677 14561 raft_consensus.cc:2454] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b36fcf88134e9f801650fa8758fc2b in term 1.
I20260620 00:20:25.516692 14293 raft_consensus.cc:2454] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b36fcf88134e9f801650fa8758fc2b in term 1.
I20260620 00:20:25.516865 14360 leader_election.cc:302] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b36fcf88134e9f801650fa8758fc2b, c1d8a755cb574d87a8a0eebd7e3e3dee; no voters:
I20260620 00:20:25.516991 14632 raft_consensus.cc:2781] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:20:25.517125 14632 raft_consensus.cc:686] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 1 LEADER]: Becoming Leader. State: Replica: 14b36fcf88134e9f801650fa8758fc2b, State: Running, Role: LEADER
I20260620 00:20:25.517263 14632 consensus_queue.cc:237] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.518105 14148 catalog_manager.cc:5495] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "14b36fcf88134e9f801650fa8758fc2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } health_report { overall_health: UNKNOWN } } }
I20260620 00:20:25.543265 14620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.540668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48128 (local address 127.0.98.2:34485)
0620 00:20:25.540723 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.540726 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.540736 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:20:25.540775 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.540776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.540797 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.540864 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:20:25.540972 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.541535 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.542432 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.542728 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.542789 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.542949 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:20:25.543061 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:20:25.543127 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.543151 (+ 24us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.543173 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:20:25.545408 14561 raft_consensus.cc:1261] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [term 1 FOLLOWER]: Refusing update from remote peer 14b36fcf88134e9f801650fa8758fc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:25.545465 14293 raft_consensus.cc:1261] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Refusing update from remote peer 14b36fcf88134e9f801650fa8758fc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:25.545837 14634 consensus_queue.cc:1035] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:25.545964 14632 consensus_queue.cc:1035] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:25.548931 14638 mvcc.cc:205] Tried to move back new op lower bound from 7298723125431132160 to 7298723125319245824. Current Snapshot: MvccSnapshot[applied={T|T < 7298723125431132160}]
I20260620 00:20:25.549899 14639 mvcc.cc:205] Tried to move back new op lower bound from 7298723125431132160 to 7298723125319245824. Current Snapshot: MvccSnapshot[applied={T|T < 7298723125431132160}]
I20260620 00:20:25.551443 14148 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:20:25.551565 14148 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:20:25.552376 14148 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Sending DeleteTablet for 3 replicas of tablet 4a5590dc6ca04fe3b52e58e5eeec819b
I20260620 00:20:25.552706 14407 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5590dc6ca04fe3b52e58e5eeec819b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48120
I20260620 00:20:25.552765 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 14210
I20260620 00:20:25.552822 14541 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5590dc6ca04fe3b52e58e5eeec819b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43912
I20260620 00:20:25.553565 14648 tablet_replica.cc:330] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: stopping tablet replica
I20260620 00:20:25.553673 14648 raft_consensus.cc:2229] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:20:25.553786 14648 raft_consensus.cc:2258] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:20:25.554243 14648 ts_tablet_manager.cc:1905] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:20:25.555541 14647 tablet_replica.cc:330] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: stopping tablet replica
I20260620 00:20:25.555678 14647 raft_consensus.cc:2229] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:20:25.555773 14648 ts_tablet_manager.cc:1918] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:20:25.555790 14647 raft_consensus.cc:2258] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:20:25.555846 14648 log.cc:1196] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/4a5590dc6ca04fe3b52e58e5eeec819b
I20260620 00:20:25.556116 14648 ts_tablet_manager.cc:1939] T 4a5590dc6ca04fe3b52e58e5eeec819b P aec66acaa4b5442f8226c144a10d3a8f: Deleting consensus metadata
I20260620 00:20:25.556265 14647 ts_tablet_manager.cc:1905] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:20:25.556564 14132 catalog_manager.cc:4841] TS aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915): tablet 4a5590dc6ca04fe3b52e58e5eeec819b (table test-table [id=3ed704e2227740aabd186bcc84949a6b]) successfully deleted
W20260620 00:20:25.557684 14132 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45347 (error 108)
I20260620 00:20:25.558056 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 14344
I20260620 00:20:25.558218 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.558075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45347 (local address 127.0.0.1:41148)
0620 00:20:25.558146 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.558182 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260620 00:20:25.558717 14132 catalog_manager.cc:4584] TS c1d8a755cb574d87a8a0eebd7e3e3dee (127.0.98.1:45347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4a5590dc6ca04fe3b52e58e5eeec819b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45347: connect: Connection refused (error 111)
I20260620 00:20:25.562943 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 14478
I20260620 00:20:25.562969 14619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.562833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34485 (local address 127.0.0.1:48140)
0620 00:20:25.562894 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.562930 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:34485: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:20:25.567432 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 14116
I20260620 00:20:25.572846 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33365
--webserver_interface=127.0.98.62
--webserver_port=42431
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:25.656941 14650 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:25.657250 14650 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:25.657299 14650 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:25.658543 14650 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:25.658589 14650 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:25.658604 14650 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:25.658617 14650 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:25.658630 14650 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:25.660164 14650 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14650
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:25.660386 14650 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:25.660566 14650 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:25.669433 14657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:25.669433 14656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:25.669585 14650 server_base.cc:1034] running on GCE node
W20260620 00:20:25.669476 14659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:25.669996 14650 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:25.670336 14650 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:25.671500 14650 hybrid_clock.cc:648] HybridClock initialized: now 1781914825671489 us; error 33 us; skew 500 ppm
Jun 20 00:20:25 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:25.678843 14650 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:20:25.679435 14650 webserver.cc:460] Webserver started at http://127.0.98.62:42431/ using document root <none> and password file <none>
I20260620 00:20:25.679713 14650 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:25.679776 14650 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:25.681489 14650 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:25.682319 14666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.682597 14650 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:25.682683 14650 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "178515fe049d475e989183850879c65e"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "35ccdfe1795e786876b6c32b7301dd91"
server_key_iv: "a370ca26dccbe09c6820dd5684979ee9"
server_key_version: "encryptionkey@0"
I20260620 00:20:25.683156 14650 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:25.709379 14650 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:25.712356 14650 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:25.712532 14650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:25.717257 14650 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33365
I20260620 00:20:25.717303 14718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33365 every 8 connection(s)
I20260620 00:20:25.717765 14650 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:20:25.718410 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 14650
I20260620 00:20:25.718847 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45347
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33365
--webserver_port=39663
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:20:25.720571 14719 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:20:25.721305 14719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Bootstrap starting.
I20260620 00:20:25.730312 14719 log.cc:824] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:25.732791 14719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:20:25.733131 14719 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Bootstrap complete.
I20260620 00:20:25.735680 14719 raft_consensus.cc:348] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:25.735934 14719 raft_consensus.cc:729] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178515fe049d475e989183850879c65e, State: Initialized, Role: FOLLOWER
I20260620 00:20:25.736168 14719 consensus_queue.cc:260] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:25.736255 14719 raft_consensus.cc:388] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:20:25.736318 14719 raft_consensus.cc:482] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:20:25.736377 14719 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:20:25.737542 14719 raft_consensus.cc:504] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:25.737639 14719 leader_election.cc:302] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 178515fe049d475e989183850879c65e; no voters:
I20260620 00:20:25.737881 14719 leader_election.cc:288] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:20:25.737911 14723 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:20:25.738042 14723 raft_consensus.cc:686] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 2 LEADER]: Becoming Leader. State: Replica: 178515fe049d475e989183850879c65e, State: Running, Role: LEADER
I20260620 00:20:25.738207 14719 sys_catalog.cc:564] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:25.738207 14723 consensus_queue.cc:237] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:25.738835 14724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "178515fe049d475e989183850879c65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } } }
I20260620 00:20:25.738854 14725 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 178515fe049d475e989183850879c65e. Latest consensus state: current_term: 2 leader_uuid: "178515fe049d475e989183850879c65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } } }
I20260620 00:20:25.739001 14724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:25.739018 14725 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:25.739293 14732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:20:25.740082 14732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:20:25.740262 14732 catalog_manager.cc:1259] Loaded cluster ID: 987fd1b0ac5042cab0f89b3bdddd68a6
I20260620 00:20:25.740495 14732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:20:25.740803 14732 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260620 00:20:25.746078 14732 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260620 00:20:25.749264 14732 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:20:25.749570 14732 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Loaded TSK: 0
I20260620 00:20:25.749867 14732 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:20:25.806598 14721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:25.806869 14721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:25.806911 14721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:25.808190 14721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:25.808265 14721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:25.808314 14721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:20:25.809875 14721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39663
--tserver_master_addrs=127.0.98.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:25.810168 14721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:25.810427 14721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:25.816618 14747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:25.816803 14746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:25.816629 14749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:25.816974 14721 server_base.cc:1034] running on GCE node
I20260620 00:20:25.817209 14721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:25.817476 14721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:25.818666 14721 hybrid_clock.cc:648] HybridClock initialized: now 1781914825818646 us; error 33 us; skew 500 ppm
Jun 20 00:20:25 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:25.821844 14721 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:20:25.822402 14721 webserver.cc:460] Webserver started at http://127.0.98.1:39663/ using document root <none> and password file <none>
I20260620 00:20:25.822651 14721 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:25.822737 14721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:25.824185 14721 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:25.824954 14756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.825156 14721 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:20:25.825250 14721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "edbcb73a595a2ecef8056dfe349f3002"
server_key_iv: "74df5ddcc5f6a748a3d3a6c915eb4499"
server_key_version: "encryptionkey@0"
I20260620 00:20:25.825639 14721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:25.840514 14721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:25.842936 14721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:25.843130 14721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:25.843515 14721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:25.844033 14763 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:20:25.846203 14721 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:20:25.846329 14721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:20:25.846381 14721 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:20:25.847059 14721 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:20:25.847118 14721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:20:25.847193 14763 tablet_bootstrap.cc:492] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Bootstrap starting.
I20260620 00:20:25.854548 14721 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45347
I20260620 00:20:25.854943 14721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:20:25.855792 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 14721
I20260620 00:20:25.856101 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34485
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33365
--webserver_port=41959
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:20:25.858075 14870 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45347 every 8 connection(s)
I20260620 00:20:25.859143 14763 log.cc:824] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:25.861986 14763 tablet_bootstrap.cc:492] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:20:25.862365 14763 tablet_bootstrap.cc:492] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Bootstrap complete.
Jun 20 00:20:25 dist-test-slave-r81b krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:25.863574 14763 ts_tablet_manager.cc:1397] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.007s
I20260620 00:20:25.865005 14763 raft_consensus.cc:348] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.865348 14763 raft_consensus.cc:729] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d8a755cb574d87a8a0eebd7e3e3dee, State: Initialized, Role: FOLLOWER
I20260620 00:20:25.865517 14763 consensus_queue.cc:260] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:25.866084 14763 ts_tablet_manager.cc:1428] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Time spent starting tablet: real 0.002s user 0.006s sys 0.000s
I20260620 00:20:25.868193 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.855721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:33891)
0620 00:20:25.856490 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.856542 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.857300 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.857972 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.857976 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.858366 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:25.858567 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.858572 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.860290 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.860291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.861093 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.861095 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.861739 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.861740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.861876 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.862448 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:25.862458 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:25.863918 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:25.866535 (+ 2617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:25.866541 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:25.866543 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:25.866776 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:25.867036 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:25.867037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:25.867037 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:25.867157 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:25.867537 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:25.867542 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:25.867682 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.867794 (+ 112us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.867974 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":709,"thread_start_us":98,"threads_started":1}
I20260620 00:20:25.868227 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.855868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33891 (local address 127.0.98.62:33365)
0620 00:20:25.857562 (+ 1694us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.857573 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.857618 (+ 45us) server_negotiation.cc:407] Connection header received
0620 00:20:25.857748 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.857752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.857793 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.857931 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:25.858693 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.860123 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.861200 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.861626 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.861759 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.864039 (+ 2280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:25.864043 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:25.864044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:25.864055 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:25.866433 (+ 2378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.866864 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.866868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.866868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.866961 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:25.867227 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:25.867228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:25.867229 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:25.867459 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:25.867558 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.867847 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.867974 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1632,"thread_start_us":88,"threads_started":1}
I20260620 00:20:25.869200 14871 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33365
I20260620 00:20:25.869294 14871 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:25.869470 14871 heartbeater.cc:511] Master 127.0.98.62:33365 requested a full tablet report, sending...
I20260620 00:20:25.870231 14683 ts_manager.cc:194] Registered new tserver with Master: c1d8a755cb574d87a8a0eebd7e3e3dee (127.0.98.1:45347)
I20260620 00:20:25.871361 14683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:33891
I20260620 00:20:25.872298 14871 heartbeater.cc:503] Master 127.0.98.62:33365 was elected leader, sending a full tablet report...
I20260620 00:20:25.875530 14879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.871059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41162 (local address 127.0.98.1:45347)
0620 00:20:25.872521 (+ 1462us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:25.872526 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:25.872535 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:20:25.872570 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:25.872572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:25.872589 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:25.872851 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:25.872896 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.873532 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.874572 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.875061 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.875164 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.875230 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:25.875416 (+ 186us) server_negotiation.cc:299] Negotiation successful
0620 00:20:25.875455 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1430,"thread_start_us":76,"threads_started":1}
I20260620 00:20:25.875536 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:25.870818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45347 (local address 127.0.0.1:41162)
0620 00:20:25.872277 (+ 1459us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:25.872315 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:25.872386 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:25.872683 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:25.872686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:25.872688 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:25.872776 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.872778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.873641 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.873642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.874476 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:25.874478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:25.875184 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:25.875185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:25.875287 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:25.875306 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 00:20:25.875398 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 00:20:25.875455 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1416,"thread_start_us":95,"threads_started":1}
I20260620 00:20:25.876287 14803 tablet_service.cc:1514] Processing DeleteTablet for tablet 4a5590dc6ca04fe3b52e58e5eeec819b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:25 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41162
I20260620 00:20:25.876829 14880 tablet_replica.cc:330] stopping tablet replica
I20260620 00:20:25.876957 14880 raft_consensus.cc:2229] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:20:25.877039 14880 raft_consensus.cc:2258] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:20:25.877370 14880 ts_tablet_manager.cc:1905] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:20:25.879467 14880 ts_tablet_manager.cc:1918] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:20:25.879559 14880 log.cc:1196] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/4a5590dc6ca04fe3b52e58e5eeec819b
I20260620 00:20:25.879799 14880 ts_tablet_manager.cc:1939] T 4a5590dc6ca04fe3b52e58e5eeec819b P c1d8a755cb574d87a8a0eebd7e3e3dee: Deleting consensus metadata
I20260620 00:20:25.880441 14667 catalog_manager.cc:4841] TS c1d8a755cb574d87a8a0eebd7e3e3dee (127.0.98.1:45347): tablet 4a5590dc6ca04fe3b52e58e5eeec819b (table test-table [id=3ed704e2227740aabd186bcc84949a6b]) successfully deleted
W20260620 00:20:25.960551 14875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:25.960806 14875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:25.960834 14875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:25.961992 14875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:25.962033 14875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:25.962060 14875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:20:25.963510 14875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41959
--tserver_master_addrs=127.0.98.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:25.963729 14875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:25.963915 14875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:25.969362 14888 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:25.969384 14885 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:25.969362 14886 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:25.969797 14875 server_base.cc:1034] running on GCE node
I20260620 00:20:25.970028 14875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:25.970350 14875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:25.971540 14875 hybrid_clock.cc:648] HybridClock initialized: now 1781914825971532 us; error 40 us; skew 500 ppm
Jun 20 00:20:25 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:25.974354 14875 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:20:25.974881 14875 webserver.cc:460] Webserver started at http://127.0.98.2:41959/ using document root <none> and password file <none>
I20260620 00:20:25.975124 14875 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:25.975211 14875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:25.976771 14875 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:20:25.977522 14895 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:25.977711 14875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:20:25.977821 14875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "14b36fcf88134e9f801650fa8758fc2b"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "6eee1e52848b85446941c0c06fadab94"
server_key_iv: "90706db93e8bf8c18d332e021dd1a60a"
server_key_version: "encryptionkey@0"
I20260620 00:20:25.978184 14875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:25.998624 14875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:26.000741 14875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:26.000918 14875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:26.001175 14875 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:26.001624 14902 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260620 00:20:26.003209 14902 ts_tablet_manager.cc:1852] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260620 00:20:26.003491 14902 ts_tablet_manager.cc:1905] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:20:26.005726 14902 ts_tablet_manager.cc:1918] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:20:26.005821 14902 log.cc:1196] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/4a5590dc6ca04fe3b52e58e5eeec819b
I20260620 00:20:26.006098 14902 ts_tablet_manager.cc:1939] T 4a5590dc6ca04fe3b52e58e5eeec819b P 14b36fcf88134e9f801650fa8758fc2b: Deleting consensus metadata
I20260620 00:20:26.006331 14875 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260620 00:20:26.006421 14875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260620 00:20:26.006475 14875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:26.006500 14875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.013237 14875 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34485
I20260620 00:20:26.013324 15009 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34485 every 8 connection(s)
I20260620 00:20:26.013760 14875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:20:26 dist-test-slave-r81b krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914825, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:26.022564 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 14875
I20260620 00:20:26.022925 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39915
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33365
--webserver_port=38059
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:20:26.023729 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.014600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37433 (local address 127.0.98.62:33365)
0620 00:20:26.014667 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:26.014669 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:26.014932 (+ 263us) server_negotiation.cc:407] Connection header received
0620 00:20:26.015525 (+ 593us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:26.015526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:26.015537 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:26.015603 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:26.016308 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.016811 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.017894 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.018100 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.018161 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.020540 (+ 2379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:26.020541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:26.020542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:26.020549 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:26.021950 (+ 1401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:26.022470 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:26.022471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:26.022472 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:26.022551 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:26.022924 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:26.022925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:26.022925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:26.023091 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:26.023161 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:26.023466 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 00:20:26.023583 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:20:26.023751 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.014598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:37433)
0620 00:20:26.014798 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.014837 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:26.015400 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:26.015664 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:26.015667 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:26.015938 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:26.016219 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.016224 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.016905 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.016906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.017807 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.017810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.018218 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.018219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.018331 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.018964 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:26.018973 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:26.020448 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:26.022049 (+ 1601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:26.022055 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:26.022057 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:26.022353 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:26.022681 (+ 328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:26.022683 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:26.022683 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:26.022849 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:26.023177 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:26.023183 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:26.023327 (+ 144us) client_negotiation.cc:769] Sending connection context
0620 00:20:26.023421 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 00:20:26.023562 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":114,"threads_started":1}
I20260620 00:20:26.024369 15010 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33365
I20260620 00:20:26.024497 15010 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:26.024693 15010 heartbeater.cc:511] Master 127.0.98.62:33365 requested a full tablet report, sending...
I20260620 00:20:26.025182 14683 ts_manager.cc:194] Registered new tserver with Master: 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485)
I20260620 00:20:26.025619 14683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37433
W20260620 00:20:26.106655 15013 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:26.106902 15013 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:26.106930 15013 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:26.108251 15013 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:26.108299 15013 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:26.108331 15013 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:20:26.109935 15013 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38059
--tserver_master_addrs=127.0.98.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15013
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:26.110219 15013 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:26.110476 15013 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:26.115886 15018 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:26.115895 15021 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:26.115891 15019 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:26.115919 15013 server_base.cc:1034] running on GCE node
I20260620 00:20:26.116425 15013 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:26.116675 15013 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:26.117821 15013 hybrid_clock.cc:648] HybridClock initialized: now 1781914826117809 us; error 27 us; skew 500 ppm
Jun 20 00:20:26 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:26.120652 15013 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:20:26.121137 15013 webserver.cc:460] Webserver started at http://127.0.98.3:38059/ using document root <none> and password file <none>
I20260620 00:20:26.121402 15013 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:26.121465 15013 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:26.122928 15013 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:26.123601 15028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.123798 15013 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:20:26.123884 15013 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "aec66acaa4b5442f8226c144a10d3a8f"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "85787f21c884e2fc88b05d3bdbe0df95"
server_key_iv: "c09f8c17d5301810c4af2a14e43e8dfd"
server_key_version: "encryptionkey@0"
I20260620 00:20:26.124343 15013 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:26.165668 15013 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:26.168085 15013 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:26.168236 15013 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:26.168432 15013 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:26.168758 15013 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:26.168797 15013 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.168849 15013 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:26.168883 15013 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.174650 15013 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39915
I20260620 00:20:26.174742 15141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39915 every 8 connection(s)
I20260620 00:20:26.175091 15013 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:20:26.179445 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 15013
Jun 20 00:20:26 dist-test-slave-r81b krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:26.185470 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.175919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53921 (local address 127.0.98.62:33365)
0620 00:20:26.175969 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:26.175971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:26.176220 (+ 249us) server_negotiation.cc:407] Connection header received
0620 00:20:26.176777 (+ 557us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:26.176778 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:26.176791 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:26.176862 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:26.177877 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.178408 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.179549 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.179858 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.179942 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.182455 (+ 2513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:26.182456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:26.182457 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:26.182465 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:26.183979 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:26.184468 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:26.184469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:26.184470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:26.184534 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:26.184789 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:26.184790 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:26.184790 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:26.184904 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:26.184980 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:26.185269 (+ 289us) server_negotiation.cc:299] Negotiation successful
0620 00:20:26.185380 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:20:26.185647 15144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.175873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:53921)
0620 00:20:26.176070 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.176112 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:26.176649 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:26.176970 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:26.176975 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:26.177560 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:26.177763 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.177768 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.178504 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.178505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.179406 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.179409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.180042 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.180043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.180241 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.180848 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:26.180860 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:26.182358 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:26.184084 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:26.184091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:26.184094 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:26.184375 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:26.184610 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:26.184611 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:26.184611 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:26.184719 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:26.184972 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:26.184978 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:26.185133 (+ 155us) client_negotiation.cc:769] Sending connection context
0620 00:20:26.185236 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 00:20:26.185403 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":109,"threads_started":1}
I20260620 00:20:26.186108 15142 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33365
I20260620 00:20:26.186226 15142 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:26.186411 15142 heartbeater.cc:511] Master 127.0.98.62:33365 requested a full tablet report, sending...
I20260620 00:20:26.186832 14683 ts_manager.cc:194] Registered new tserver with Master: aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915)
I20260620 00:20:26.187258 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.179956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41666 (local address 127.0.98.62:33365)
0620 00:20:26.180066 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:26.180068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:26.180105 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:20:26.180291 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:26.180292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:26.180306 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:26.180368 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:26.181081 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.181717 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.182616 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.182828 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.182912 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.184042 (+ 1130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:26.184043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:26.184044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:26.184052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:26.185712 (+ 1660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:26.186306 (+ 594us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:26.186307 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:26.186308 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:26.186365 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:26.186597 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:26.186598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:26.186598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:26.186715 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:26.186897 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:26.186935 (+ 38us) server_negotiation.cc:299] Negotiation successful
0620 00:20:26.187017 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":57,"threads_started":1}
I20260620 00:20:26.187258 14683 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53921
I20260620 00:20:26.188095 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:20:26.190761 15146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.188430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41680 (local address 127.0.98.62:33365)
0620 00:20:26.188466 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:26.188467 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:26.188585 (+ 118us) server_negotiation.cc:407] Connection header received
0620 00:20:26.188638 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:26.188639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:26.188668 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:26.188727 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:20:26.188820 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.189262 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.190037 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.190296 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.190361 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.190499 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:20:26.190590 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:20:26.190647 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:26.190690 (+ 43us) server_negotiation.cc:299] Negotiation successful
0620 00:20:26.190710 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 00:20:26.191072 14683 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:41680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:20:26.192857 14683 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:20:26.195609 14803 tablet_service.cc:1467] Processing CreateTablet for tablet c19bc36dcfc24cf085736cd426fe8013 (DEFAULT_TABLE table=test-table [id=1905d7b51f0e403eb99447ce717819a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:26.195819 14803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19bc36dcfc24cf085736cd426fe8013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:26.197423 14763 tablet_bootstrap.cc:492] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Bootstrap starting.
I20260620 00:20:26.197780 14763 tablet_bootstrap.cc:654] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:26.198400 14763 tablet_bootstrap.cc:492] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: No bootstrap required, opened a new log
I20260620 00:20:26.198472 14763 ts_tablet_manager.cc:1397] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:26.198601 14763 raft_consensus.cc:348] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.198688 14763 raft_consensus.cc:374] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:26.198748 14763 raft_consensus.cc:729] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d8a755cb574d87a8a0eebd7e3e3dee, State: Initialized, Role: FOLLOWER
I20260620 00:20:26.198799 14763 consensus_queue.cc:260] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.198872 14878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.194894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39915 (local address 127.0.0.1:43920)
0620 00:20:26.195017 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.195028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:26.195071 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:26.195585 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:26.195586 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:26.195587 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:26.195642 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.195644 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.196817 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.196818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.197643 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.197646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.198609 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.198611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.198671 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.198676 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:20:26.198744 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 00:20:26.198795 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106}
I20260620 00:20:26.198906 14763 ts_tablet_manager.cc:1428] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.199200 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.195200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43920 (local address 127.0.98.3:39915)
0620 00:20:26.195377 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:26.195380 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:26.195389 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:20:26.195442 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:26.195444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:26.195461 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:26.195581 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:26.195748 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.196693 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.197742 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.198481 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.198589 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.198654 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:26.199054 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 00:20:26.199101 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":101,"threads_started":1}
I20260620 00:20:26.199728 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.194894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34485 (local address 127.0.0.1:48152)
0620 00:20:26.195460 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.195468 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:26.195544 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:26.195943 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:26.195944 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:26.195946 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:26.195984 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.195986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.197251 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.197253 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.198486 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.198489 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.199547 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.199548 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.199596 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.199599 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:20:26.199648 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 00:20:26.199678 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":554,"mutex_wait_us":23,"thread_start_us":59,"threads_started":1}
I20260620 00:20:26.199817 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.195029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48152 (local address 127.0.98.2:34485)
0620 00:20:26.195245 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:26.195250 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:26.195538 (+ 288us) server_negotiation.cc:407] Connection header received
0620 00:20:26.195675 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:26.195677 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:26.195695 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:26.195874 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:26.196079 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.197132 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.198579 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.199379 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.199450 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.199508 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:26.199683 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:20:26.199731 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":75,"threads_started":1}
I20260620 00:20:26.200011 15076 tablet_service.cc:1467] Processing CreateTablet for tablet c19bc36dcfc24cf085736cd426fe8013 (DEFAULT_TABLE table=test-table [id=1905d7b51f0e403eb99447ce717819a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:26.200405 15076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19bc36dcfc24cf085736cd426fe8013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:26.200871 14944 tablet_service.cc:1467] Processing CreateTablet for tablet c19bc36dcfc24cf085736cd426fe8013 (DEFAULT_TABLE table=test-table [id=1905d7b51f0e403eb99447ce717819a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:26.201154 14944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c19bc36dcfc24cf085736cd426fe8013. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:26.202863 14902 tablet_bootstrap.cc:492] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: Bootstrap starting.
I20260620 00:20:26.203446 15152 tablet_bootstrap.cc:492] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: Bootstrap starting.
I20260620 00:20:26.203766 14902 tablet_bootstrap.cc:654] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:26.204085 14902 log.cc:824] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:26.204089 15152 tablet_bootstrap.cc:654] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:26.204671 15152 log.cc:824] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:26.204928 14902 tablet_bootstrap.cc:492] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: No bootstrap required, opened a new log
I20260620 00:20:26.204981 14902 ts_tablet_manager.cc:1397] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:20:26.205498 15152 tablet_bootstrap.cc:492] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: No bootstrap required, opened a new log
I20260620 00:20:26.205569 15152 ts_tablet_manager.cc:1397] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:20:26.207108 14902 raft_consensus.cc:348] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.207309 14902 raft_consensus.cc:374] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:26.207355 14902 raft_consensus.cc:729] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14b36fcf88134e9f801650fa8758fc2b, State: Initialized, Role: FOLLOWER
I20260620 00:20:26.207484 14902 consensus_queue.cc:260] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.207612 15152 raft_consensus.cc:348] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.207798 15152 raft_consensus.cc:374] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:26.207820 14902 ts_tablet_manager.cc:1428] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:20:26.207882 15010 heartbeater.cc:503] Master 127.0.98.62:33365 was elected leader, sending a full tablet report...
I20260620 00:20:26.207849 15152 raft_consensus.cc:729] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec66acaa4b5442f8226c144a10d3a8f, State: Initialized, Role: FOLLOWER
I20260620 00:20:26.208073 15152 consensus_queue.cc:260] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.208483 15152 ts_tablet_manager.cc:1428] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:20:26.208621 15142 heartbeater.cc:503] Master 127.0.98.62:33365 was elected leader, sending a full tablet report...
W20260620 00:20:26.264354 15011 tablet.cc:2369] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:20:26.358769 14872 tablet.cc:2369] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:20:26.425716 15143 tablet.cc:2369] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:20:26.484350 15155 raft_consensus.cc:482] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:26.484586 15155 raft_consensus.cc:504] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.485019 15155 leader_election.cc:288] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915), c1d8a755cb574d87a8a0eebd7e3e3dee (127.0.98.1:45347)
I20260620 00:20:26.488473 15012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.485323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39915 (local address 127.0.98.2:53785)
0620 00:20:26.485388 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.485400 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:26.485445 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:26.485724 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:26.485726 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:26.485727 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:26.485809 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.485811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.486546 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.486547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.487440 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.487443 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.488176 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.488177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.488278 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.488282 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:20:26.488346 (+ 64us) client_negotiation.cc:240] Negotiation successful
0620 00:20:26.488388 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:20:26.488494 15150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.485480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53785 (local address 127.0.98.3:39915)
0620 00:20:26.485542 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:26.485544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:26.485590 (+ 46us) server_negotiation.cc:407] Connection header received
0620 00:20:26.485621 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:26.485623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:26.485636 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:26.485716 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:26.485907 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.486465 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.487549 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.488067 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.488122 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.488178 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:26.488394 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 00:20:26.488443 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260620 00:20:26.488809 15157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.485455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45347 (local address 127.0.98.2:38343)
0620 00:20:26.485586 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.485598 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:26.485730 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:26.485906 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:26.485907 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:26.485909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:26.485978 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.485979 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.486874 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.486875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.487775 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:26.487778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.488549 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.488550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:26.488632 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.488638 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:20:26.488707 (+ 69us) client_negotiation.cc:240] Negotiation successful
0620 00:20:26.488747 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":56,"threads_started":1}
I20260620 00:20:26.489046 15158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.485597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38343 (local address 127.0.98.1:45347)
0620 00:20:26.485749 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:26.485751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:26.485758 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:20:26.485803 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:26.485803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:26.485814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:26.485888 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:26.486080 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.486783 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.487865 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.488467 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.488546 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.488560 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:26.488921 (+ 361us) server_negotiation.cc:299] Negotiation successful
0620 00:20:26.488977 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":89,"threads_started":1}
I20260620 00:20:26.489017 15096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19bc36dcfc24cf085736cd426fe8013" candidate_uuid: "14b36fcf88134e9f801650fa8758fc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec66acaa4b5442f8226c144a10d3a8f" is_pre_election: true
I20260620 00:20:26.489218 15096 raft_consensus.cc:2454] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b36fcf88134e9f801650fa8758fc2b in term 0.
I20260620 00:20:26.489306 14825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19bc36dcfc24cf085736cd426fe8013" candidate_uuid: "14b36fcf88134e9f801650fa8758fc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" is_pre_election: true
I20260620 00:20:26.489473 14825 raft_consensus.cc:2454] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 14b36fcf88134e9f801650fa8758fc2b in term 0.
I20260620 00:20:26.489485 14896 leader_election.cc:302] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b36fcf88134e9f801650fa8758fc2b, aec66acaa4b5442f8226c144a10d3a8f; no voters:
I20260620 00:20:26.489677 15155 raft_consensus.cc:2781] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:20:26.489737 15155 raft_consensus.cc:482] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:20:26.489753 15155 raft_consensus.cc:3037] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:26.490437 15155 raft_consensus.cc:504] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.490573 15155 leader_election.cc:288] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [CANDIDATE]: Term 1 election: Requested vote from peers aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915), c1d8a755cb574d87a8a0eebd7e3e3dee (127.0.98.1:45347)
I20260620 00:20:26.490731 15096 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19bc36dcfc24cf085736cd426fe8013" candidate_uuid: "14b36fcf88134e9f801650fa8758fc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec66acaa4b5442f8226c144a10d3a8f"
I20260620 00:20:26.490751 14825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c19bc36dcfc24cf085736cd426fe8013" candidate_uuid: "14b36fcf88134e9f801650fa8758fc2b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee"
I20260620 00:20:26.490836 15096 raft_consensus.cc:3037] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:26.490833 14825 raft_consensus.cc:3037] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:26.491497 14825 raft_consensus.cc:2454] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b36fcf88134e9f801650fa8758fc2b in term 1.
I20260620 00:20:26.491600 15096 raft_consensus.cc:2454] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 14b36fcf88134e9f801650fa8758fc2b in term 1.
I20260620 00:20:26.491657 14896 leader_election.cc:302] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 14b36fcf88134e9f801650fa8758fc2b, c1d8a755cb574d87a8a0eebd7e3e3dee; no voters:
I20260620 00:20:26.491765 15155 raft_consensus.cc:2781] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:20:26.491935 15155 raft_consensus.cc:686] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 1 LEADER]: Becoming Leader. State: Replica: 14b36fcf88134e9f801650fa8758fc2b, State: Running, Role: LEADER
I20260620 00:20:26.492036 15155 consensus_queue.cc:237] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.492919 14682 catalog_manager.cc:5495] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b reported cstate change: term changed from 0 to 1, leader changed from <none> to 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2). New cstate: current_term: 1 leader_uuid: "14b36fcf88134e9f801650fa8758fc2b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } health_report { overall_health: UNKNOWN } } }
I20260620 00:20:26.558837 15149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.556650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48158 (local address 127.0.98.2:34485)
0620 00:20:26.556718 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:26.556719 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:26.556727 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:20:26.556763 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:26.556764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:26.556786 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:26.556850 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:20:26.556966 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.557442 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.558149 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:26.558339 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:26.558389 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:26.558535 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:20:26.558636 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:20:26.558696 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:26.558736 (+ 40us) server_negotiation.cc:299] Negotiation successful
0620 00:20:26.558753 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:20:26.560706 14825 raft_consensus.cc:1261] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Refusing update from remote peer 14b36fcf88134e9f801650fa8758fc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:26.560740 15096 raft_consensus.cc:1261] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [term 1 FOLLOWER]: Refusing update from remote peer 14b36fcf88134e9f801650fa8758fc2b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:26.561134 15159 consensus_queue.cc:1035] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:26.561297 15155 consensus_queue.cc:1035] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:26.563707 15162 mvcc.cc:205] Tried to move back new op lower bound from 7298723129589817344 to 7298723129312124928. Current Snapshot: MvccSnapshot[applied={T|T < 7298723129589817344}]
I20260620 00:20:26.564114 15164 mvcc.cc:205] Tried to move back new op lower bound from 7298723129589817344 to 7298723129312124928. Current Snapshot: MvccSnapshot[applied={T|T < 7298723129589817344 or (T in {7298723129589817344})}]
I20260620 00:20:26.565845 14683 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:41680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:20:26.565946 14683 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:41680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:20:26.566799 14683 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Sending DeleteTablet for 3 replicas of tablet c19bc36dcfc24cf085736cd426fe8013
I20260620 00:20:26.567137 14944 tablet_service.cc:1514] Processing DeleteTablet for tablet c19bc36dcfc24cf085736cd426fe8013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48152
I20260620 00:20:26.567171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 14721
I20260620 00:20:26.567684 15076 tablet_service.cc:1514] Processing DeleteTablet for tablet c19bc36dcfc24cf085736cd426fe8013 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:26 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43920
I20260620 00:20:26.567943 15171 tablet_replica.cc:330] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: stopping tablet replica
I20260620 00:20:26.568073 15171 raft_consensus.cc:2229] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:20:26.568113 15170 tablet_replica.cc:330] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: stopping tablet replica
I20260620 00:20:26.568243 15171 raft_consensus.cc:2258] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:20:26.568274 15170 raft_consensus.cc:2229] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:20:26.568369 15170 raft_consensus.cc:2258] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:20:26.568730 15170 ts_tablet_manager.cc:1905] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:20:26.570374 15170 ts_tablet_manager.cc:1918] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:20:26.570434 15170 log.cc:1196] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/c19bc36dcfc24cf085736cd426fe8013
I20260620 00:20:26.570645 15170 ts_tablet_manager.cc:1939] T c19bc36dcfc24cf085736cd426fe8013 P 14b36fcf88134e9f801650fa8758fc2b: Deleting consensus metadata
I20260620 00:20:26.571029 14670 catalog_manager.cc:4841] TS 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485): tablet c19bc36dcfc24cf085736cd426fe8013 (table test-table [id=1905d7b51f0e403eb99447ce717819a3]) successfully deleted
I20260620 00:20:26.572662 15171 ts_tablet_manager.cc:1905] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260620 00:20:26.573347 14667 connection.cc:535] client connection to 127.0.98.1:45347 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260620 00:20:26.573474 14667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:20:26.573931 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 14875
I20260620 00:20:26.574698 15171 ts_tablet_manager.cc:1918] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:20:26.574782 15171 log.cc:1196] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/c19bc36dcfc24cf085736cd426fe8013
I20260620 00:20:26.575059 15171 ts_tablet_manager.cc:1939] T c19bc36dcfc24cf085736cd426fe8013 P aec66acaa4b5442f8226c144a10d3a8f: Deleting consensus metadata
I20260620 00:20:26.576648 14667 catalog_manager.cc:4841] TS aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915): tablet c19bc36dcfc24cf085736cd426fe8013 (table test-table [id=1905d7b51f0e403eb99447ce717819a3]) successfully deleted
I20260620 00:20:26.576790 15148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.576591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45347 (local address 127.0.0.1:41170)
0620 00:20:26.576661 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.576748 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260620 00:20:26.577003 14667 catalog_manager.cc:4584] TS c1d8a755cb574d87a8a0eebd7e3e3dee (127.0.98.1:45347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c19bc36dcfc24cf085736cd426fe8013: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45347: connect: Connection refused (error 111)
I20260620 00:20:26.580430 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 15013
I20260620 00:20:26.585544 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 14650
I20260620 00:20:26.591015 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33365
--webserver_interface=127.0.98.62
--webserver_port=42431
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33365
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:26.675251 15173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:26.675527 15173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:26.675565 15173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:26.676824 15173 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:26.676913 15173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:26.676936 15173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:26.676955 15173 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:26.676982 15173 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:26.678624 15173 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42431
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:26.678941 15173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:26.679194 15173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:26.684458 15178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:26.684636 15181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:26.684762 15173 server_base.cc:1034] running on GCE node
W20260620 00:20:26.684507 15179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:26.685078 15173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:26.685354 15173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:26.686502 15173 hybrid_clock.cc:648] HybridClock initialized: now 1781914826686485 us; error 32 us; skew 500 ppm
Jun 20 00:20:26 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:26.689294 15173 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:20:26.689852 15173 webserver.cc:460] Webserver started at http://127.0.98.62:42431/ using document root <none> and password file <none>
I20260620 00:20:26.690100 15173 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:26.690187 15173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:26.691605 15173 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:26.692412 15188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.692719 15173 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:26.692844 15173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "178515fe049d475e989183850879c65e"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "35ccdfe1795e786876b6c32b7301dd91"
server_key_iv: "a370ca26dccbe09c6820dd5684979ee9"
server_key_version: "encryptionkey@0"
I20260620 00:20:26.693266 15173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:26.703419 15173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:26.705503 15173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:26.705685 15173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:26.710093 15173 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33365
I20260620 00:20:26.710129 15240 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33365 every 8 connection(s)
I20260620 00:20:26.710562 15173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:20:26.712194 15241 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:20:26.712920 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Bootstrap starting.
I20260620 00:20:26.716236 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 15173
I20260620 00:20:26.716571 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:45347
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:33365
--webserver_port=39663
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:20:26.721627 15241 log.cc:824] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:26.724012 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:20:26.724385 15241 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e: Bootstrap complete.
I20260620 00:20:26.726675 15241 raft_consensus.cc:348] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:26.726943 15241 raft_consensus.cc:729] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 178515fe049d475e989183850879c65e, State: Initialized, Role: FOLLOWER
I20260620 00:20:26.727151 15241 consensus_queue.cc:260] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:26.727252 15241 raft_consensus.cc:388] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:20:26.727305 15241 raft_consensus.cc:482] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:20:26.727377 15241 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 2 FOLLOWER]: Advancing to term 3
I20260620 00:20:26.728518 15241 raft_consensus.cc:504] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:26.728647 15241 leader_election.cc:302] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 178515fe049d475e989183850879c65e; no voters:
I20260620 00:20:26.728865 15241 leader_election.cc:288] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 00:20:26.728902 15245 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 3 FOLLOWER]: Leader election won for term 3
I20260620 00:20:26.729030 15245 raft_consensus.cc:686] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [term 3 LEADER]: Becoming Leader. State: Replica: 178515fe049d475e989183850879c65e, State: Running, Role: LEADER
I20260620 00:20:26.729121 15245 consensus_queue.cc:237] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } }
I20260620 00:20:26.729153 15241 sys_catalog.cc:564] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:26.729758 15246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "178515fe049d475e989183850879c65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } } }
I20260620 00:20:26.729760 15247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 178515fe049d475e989183850879c65e. Latest consensus state: current_term: 3 leader_uuid: "178515fe049d475e989183850879c65e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "178515fe049d475e989183850879c65e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33365 } } }
I20260620 00:20:26.729887 15246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:26.729890 15247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 178515fe049d475e989183850879c65e [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:26.730082 15254 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:20:26.730890 15254 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:20:26.730989 15254 catalog_manager.cc:1259] Loaded cluster ID: 987fd1b0ac5042cab0f89b3bdddd68a6
I20260620 00:20:26.731014 15254 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260620 00:20:26.731225 15254 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f9b3dac69ad google::LogMessage::Fail() at ??:0
@ 0x7f9b3dac988a google::LogMessage::SendToLog() at ??:0
@ 0x7f9b3dac64de google::LogMessage::Flush() at ??:0
@ 0x7f9b3dac7a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f9b459eb0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f9b459f48b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f9b3f9940d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f9b3f989954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f9b407ed6db start_thread at ??:0
@ 0x7f9b3c5c461f clone at ??:0
W20260620 00:20:26.814973 15243 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:26.815236 15243 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:26.815277 15243 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:26.816684 15243 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:26.816751 15243 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:26.816799 15243 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:20:26.818440 15243 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:45347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=39663
--tserver_master_addrs=127.0.98.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15243
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:26.818730 15243 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:26.818967 15243 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:26.824165 15280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:26.824419 15243 server_base.cc:1034] running on GCE node
W20260620 00:20:26.824579 15278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:26.824613 15277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:26.824927 15243 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:26.825241 15243 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:26.826449 15243 hybrid_clock.cc:648] HybridClock initialized: now 1781914826826424 us; error 41 us; skew 500 ppm
Jun 20 00:20:26 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:26.829541 15243 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:20:26.830063 15243 webserver.cc:460] Webserver started at http://127.0.98.1:39663/ using document root <none> and password file <none>
I20260620 00:20:26.830341 15243 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:26.830395 15243 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:26.832031 15243 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:20:26.832885 15287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.833066 15243 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:20:26.833151 15243 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee"
format_stamp: "Formatted at 2026-06-20 00:20:24 on dist-test-slave-r81b"
server_key: "edbcb73a595a2ecef8056dfe349f3002"
server_key_iv: "74df5ddcc5f6a748a3d3a6c915eb4499"
server_key_version: "encryptionkey@0"
I20260620 00:20:26.833519 15243 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:26.843891 15243 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:26.846450 15243 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:26.846617 15243 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:26.846901 15243 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:26.847378 15294 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:20:26.849406 15243 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:20:26.849486 15243 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:20:26.849539 15243 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:20:26.850241 15243 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:20:26.850301 15243 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:20:26.850386 15294 tablet_bootstrap.cc:492] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Bootstrap starting.
I20260620 00:20:26.857656 15243 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45347
I20260620 00:20:26.857745 15401 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45347 every 8 connection(s)
I20260620 00:20:26.858124 15243 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:20:26.861483 15294 log.cc:824] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:26.862588 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 15243
I20260620 00:20:26.862919 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:34485
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:33365
--webserver_port=41959
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:20:26.864245 15294 tablet_bootstrap.cc:492] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:20:26.864564 15294 tablet_bootstrap.cc:492] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Bootstrap complete.
I20260620 00:20:26.865808 15294 ts_tablet_manager.cc:1397] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.008s
I20260620 00:20:26.867424 15294 raft_consensus.cc:348] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.867765 15294 raft_consensus.cc:729] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1d8a755cb574d87a8a0eebd7e3e3dee, State: Initialized, Role: FOLLOWER
I20260620 00:20:26.867945 15294 consensus_queue.cc:260] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:26.868299 15294 ts_tablet_manager.cc:1428] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
W20260620 00:20:26.943007 15404 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:20:26.858837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:33923)
0620 00:20:26.859013 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.859059 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:26.859515 (+ 456us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:26.942775 (+ 83260us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":97,"threads_started":1}
W20260620 00:20:26.943357 15402 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33365 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33365: Client connection negotiation failed: client connection to 127.0.98.62:33365: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:20:26.943811 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.943661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:33063)
0620 00:20:26.943732 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.943781 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260620 00:20:26.944335 15404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.944205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:36979)
0620 00:20:26.944280 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.944310 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260620 00:20:26.944470 15402 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:20:26.952422 15406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:26.952623 15406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:26.952647 15406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:26.953871 15406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:26.953917 15406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:26.953944 15406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:20:26.955379 15406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:34485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=41959
--tserver_master_addrs=127.0.98.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:26.955574 15406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:26.955750 15406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:26.960924 15415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:26.960942 15412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:26.960919 15413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:26.961333 15406 server_base.cc:1034] running on GCE node
I20260620 00:20:26.961540 15406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:26.961762 15406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:26.962909 15406 hybrid_clock.cc:648] HybridClock initialized: now 1781914826962896 us; error 28 us; skew 500 ppm
Jun 20 00:20:26 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:26.970675 15406 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:20:26.971164 15406 webserver.cc:460] Webserver started at http://127.0.98.2:41959/ using document root <none> and password file <none>
I20260620 00:20:26.971388 15406 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:26.971437 15406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:26.973022 15406 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:26.973748 15422 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.973946 15406 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:26.974069 15406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "14b36fcf88134e9f801650fa8758fc2b"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "6eee1e52848b85446941c0c06fadab94"
server_key_iv: "90706db93e8bf8c18d332e021dd1a60a"
server_key_version: "encryptionkey@0"
I20260620 00:20:26.974503 15406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:26.983585 15406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:26.986092 15406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:26.986268 15406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:26.986515 15406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:26.986836 15406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:26.986899 15406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.986948 15406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:26.986981 15406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:26.993638 15406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:34485
I20260620 00:20:26.993688 15535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:34485 every 8 connection(s)
I20260620 00:20:26.994069 15406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:20:26.995198 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.994733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:34983)
0620 00:20:26.994911 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.994986 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
W20260620 00:20:26.995641 15536 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33365 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33365: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
I20260620 00:20:26.995980 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.995850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:53753)
0620 00:20:26.995923 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.995956 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260620 00:20:26.996311 15538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:26.996243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:54357)
0620 00:20:26.996269 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:26.996294 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260620 00:20:26.996444 15536 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:20:26.999241 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 15406
I20260620 00:20:26.999506 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:39915
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:33365
--webserver_port=38059
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:27.083741 15539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:27.083967 15539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:27.083993 15539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:27.085247 15539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:27.085296 15539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:27.085325 15539 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:20:27.086840 15539 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:39915
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=38059
--tserver_master_addrs=127.0.98.62:33365
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:27.087107 15539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:27.087366 15539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:27.092650 15547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:27.092730 15544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:27.092650 15545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:27.092877 15539 server_base.cc:1034] running on GCE node
I20260620 00:20:27.093104 15539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:27.093400 15539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:27.094578 15539 hybrid_clock.cc:648] HybridClock initialized: now 1781914827094562 us; error 30 us; skew 500 ppm
Jun 20 00:20:27 dist-test-slave-r81b krb5kdc[14103](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:27.097445 15539 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:20:27.097952 15539 webserver.cc:460] Webserver started at http://127.0.98.3:38059/ using document root <none> and password file <none>
I20260620 00:20:27.098222 15539 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:27.098273 15539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:27.099857 15539 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:20:27.100579 15554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:27.100775 15539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:20:27.100903 15539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "aec66acaa4b5442f8226c144a10d3a8f"
format_stamp: "Formatted at 2026-06-20 00:20:25 on dist-test-slave-r81b"
server_key: "85787f21c884e2fc88b05d3bdbe0df95"
server_key_iv: "c09f8c17d5301810c4af2a14e43e8dfd"
server_key_version: "encryptionkey@0"
I20260620 00:20:27.101918 15539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:27.128587 15539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:27.131737 15539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:27.131904 15539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:27.132210 15539 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:27.132651 15539 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:27.132699 15539 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:27.132736 15539 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:27.132759 15539 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:27.140498 15539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39915
I20260620 00:20:27.140545 15667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39915 every 8 connection(s)
I20260620 00:20:27.141042 15539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:20:27.142377 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:27.141863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:56177)
0620 00:20:27.142060 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:27.142166 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
W20260620 00:20:27.142922 15668 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:33365 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:33365: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
I20260620 00:20:27.143257 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:27.143134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:55065)
0620 00:20:27.143195 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:27.143230 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 00:20:27.143612 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:27.143483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:54291)
0620 00:20:27.143532 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:27.143584 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:20:27.143754 15668 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:20:27.145176 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 15539
I20260620 00:20:27.945103 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:27.944824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:43105)
0620 00:20:27.944993 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:27.945035 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":67,"threads_started":1}
I20260620 00:20:27.996985 15673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:27.996761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:46385)
0620 00:20:27.996891 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:27.996928 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":60,"threads_started":1}
I20260620 00:20:28.109194 15674 raft_consensus.cc:482] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:28.109333 15674 raft_consensus.cc:504] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:28.109747 15674 leader_election.cc:288] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485), aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915)
Jun 20 00:20:28 dist-test-slave-r81b krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 00:20:28 dist-test-slave-r81b krb5kdc[14103](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914826, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 00:20:28.123668 15675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:28.109964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:34485 (local address 127.0.98.1:49881)
0620 00:20:28.110362 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:28.110374 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:28.110460 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:28.111166 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:28.111167 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:28.111589 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:28.111776 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:28.111783 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:28.113055 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:28.113056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:28.114293 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:28.114295 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:28.115195 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:28.115196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:28.115398 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:28.116854 (+ 1456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:28.116866 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:28.118864 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:28.121312 (+ 2448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:28.121322 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:28.121324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:28.121782 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:28.122190 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:28.122190 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:28.122191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:28.122352 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:28.122931 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:28.122940 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:28.123119 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 00:20:28.123268 (+ 149us) client_negotiation.cc:240] Negotiation successful
0620 00:20:28.123534 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":83,"thread_start_us":69,"threads_started":1}
I20260620 00:20:28.123710 15677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:28.110207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49881 (local address 127.0.98.2:34485)
0620 00:20:28.110382 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:28.110390 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:28.110444 (+ 54us) server_negotiation.cc:407] Connection header received
0620 00:20:28.110927 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:28.110930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:28.110965 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:28.111124 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:28.111910 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:28.112953 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:28.114423 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:28.115079 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:28.115202 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:28.119003 (+ 3801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:28.119012 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:28.119012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:28.119031 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:28.121204 (+ 2173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:28.121932 (+ 728us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:28.121936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:28.121936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:28.122072 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:28.122445 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:28.122447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:28.122447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:28.122825 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:28.122923 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:28.123395 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 00:20:28.123561 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260620 00:20:28.123816 15676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:28.110128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45231 (local address 127.0.98.3:39915)
0620 00:20:28.110317 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:28.110331 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:28.110358 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:20:28.110890 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:28.110893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:28.110956 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:28.111124 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:28.111946 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:28.113249 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:28.114423 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:28.115079 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:28.115201 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:28.119313 (+ 4112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:28.119318 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:28.119318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:28.119333 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:28.121289 (+ 1956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:28.121932 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:28.121936 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:28.121936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:28.122072 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:28.122445 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:28.122447 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:28.122447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:28.122825 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:28.122923 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:28.123395 (+ 472us) server_negotiation.cc:299] Negotiation successful
0620 00:20:28.123562 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":123,"threads_started":1}
I20260620 00:20:28.124380 15672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:28.109964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39915 (local address 127.0.98.1:45231)
0620 00:20:28.110029 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:28.110042 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:28.110096 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:28.111155 (+ 1059us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:28.111158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:28.111588 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:28.111776 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:28.111782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:28.113336 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:28.113337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:28.114293 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:28.114304 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:28.115193 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:28.115194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:28.115398 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:28.116854 (+ 1456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:28.116866 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:28.119191 (+ 2325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:28.121371 (+ 2180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:28.121372 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:28.121372 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:28.121782 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:28.122189 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:28.122190 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:28.122191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:28.122352 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:28.122931 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:28.122940 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:28.123119 (+ 179us) client_negotiation.cc:769] Sending connection context
0620 00:20:28.123272 (+ 153us) client_negotiation.cc:240] Negotiation successful
0620 00:20:28.124257 (+ 985us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44,"mutex_wait_us":260}
W20260620 00:20:28.124930 15291 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
W20260620 00:20:28.125161 15288 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
I20260620 00:20:28.125227 15288 leader_election.cc:302] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d8a755cb574d87a8a0eebd7e3e3dee; no voters: 14b36fcf88134e9f801650fa8758fc2b, aec66acaa4b5442f8226c144a10d3a8f
I20260620 00:20:28.125415 15674 raft_consensus.cc:2726] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:20:28.144481 15678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:28.144090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:39733)
0620 00:20:28.144325 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:28.144398 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":155,"threads_started":1}
I20260620 00:20:28.946125 15679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:28.945630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:46369)
0620 00:20:28.945807 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:28.946032 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":81,"threads_started":1}
I20260620 00:20:28.997695 15680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:28.997465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:34045)
0620 00:20:28.997596 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:28.997651 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":73,"threads_started":1}
I20260620 00:20:29.145414 15681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:29.145149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:45195)
0620 00:20:29.145323 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:29.145367 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":64,"threads_started":1}
I20260620 00:20:29.898526 15682 raft_consensus.cc:482] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:29.898633 15682 raft_consensus.cc:504] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:29.898782 15682 leader_election.cc:288] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485), aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915)
W20260620 00:20:29.899245 15288 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
W20260620 00:20:29.899360 15291 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
I20260620 00:20:29.899417 15291 leader_election.cc:302] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d8a755cb574d87a8a0eebd7e3e3dee; no voters: 14b36fcf88134e9f801650fa8758fc2b, aec66acaa4b5442f8226c144a10d3a8f
I20260620 00:20:29.899498 15682 raft_consensus.cc:2726] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:20:29.947053 15683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:29.946777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:44937)
0620 00:20:29.946962 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:29.947004 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":72,"threads_started":1}
I20260620 00:20:29.998447 15684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:29.998224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:36555)
0620 00:20:29.998361 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:29.998401 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260620 00:20:30.146409 15685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:30.146115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:40717)
0620 00:20:30.146330 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:30.146361 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":110,"threads_started":1}
I20260620 00:20:30.947886 15686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:30.947641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:39881)
0620 00:20:30.947802 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:30.947840 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260620 00:20:30.999373 15687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:30.999091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:52899)
0620 00:20:30.999285 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:30.999327 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":99,"threads_started":1}
I20260620 00:20:31.147296 15688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:31.147022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:58997)
0620 00:20:31.147192 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:31.147245 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":77,"threads_started":1}
I20260620 00:20:31.948652 15689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:31.948425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:59807)
0620 00:20:31.948568 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:31.948606 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":63,"threads_started":1}
I20260620 00:20:32.000125 15690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:31.999901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:48273)
0620 00:20:32.000042 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:32.000080 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
I20260620 00:20:32.148113 15691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:32.147878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:33199)
0620 00:20:32.148026 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:32.148067 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260620 00:20:32.949568 15692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:32.949299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:54159)
0620 00:20:32.949475 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:32.949519 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":70,"threads_started":1}
I20260620 00:20:33.000952 15693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:33.000683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:49459)
0620 00:20:33.000858 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:33.000902 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":76,"threads_started":1}
I20260620 00:20:33.148867 15694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:33.148626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:60823)
0620 00:20:33.148765 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:33.148814 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260620 00:20:33.165292 15695 raft_consensus.cc:482] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:33.165401 15695 raft_consensus.cc:504] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:33.165594 15695 leader_election.cc:288] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485), aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915)
W20260620 00:20:33.166007 15288 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
W20260620 00:20:33.166090 15291 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
I20260620 00:20:33.166153 15291 leader_election.cc:302] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d8a755cb574d87a8a0eebd7e3e3dee; no voters: 14b36fcf88134e9f801650fa8758fc2b, aec66acaa4b5442f8226c144a10d3a8f
I20260620 00:20:33.166229 15695 raft_consensus.cc:2726] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:20:33.950533 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:33.950274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:44881)
0620 00:20:33.950435 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:33.950483 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":70,"threads_started":1}
I20260620 00:20:34.001856 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:34.001605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:50653)
0620 00:20:34.001761 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:34.001807 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":67,"threads_started":1}
I20260620 00:20:34.149686 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:34.149461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:45521)
0620 00:20:34.149593 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:34.149638 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":67,"threads_started":1}
I20260620 00:20:34.951525 15699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:34.951255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:54297)
0620 00:20:34.951423 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:34.951470 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":71,"threads_started":1}
I20260620 00:20:35.003012 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:35.002682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:51241)
0620 00:20:35.002900 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:35.002957 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":91,"threads_started":1}
I20260620 00:20:35.150575 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:35.150261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:33067)
0620 00:20:35.150436 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:35.150480 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":103,"threads_started":1}
I20260620 00:20:35.952379 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:35.952106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:40633)
0620 00:20:35.952285 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:35.952327 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":113,"threads_started":1}
I20260620 00:20:36.003932 15703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:36.003673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:54537)
0620 00:20:36.003845 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:36.003884 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":79,"threads_started":1}
I20260620 00:20:36.151584 15704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:36.151312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:45911)
0620 00:20:36.151480 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:36.151538 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":65,"threads_started":1}
I20260620 00:20:36.953197 15705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:36.952876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:35627)
0620 00:20:36.953104 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:36.953147 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":83,"threads_started":1}
I20260620 00:20:37.004881 15706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:37.004564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:48141)
0620 00:20:37.004719 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:37.004768 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":66,"threads_started":1}
I20260620 00:20:37.152494 15707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:37.152217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:42669)
0620 00:20:37.152399 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:37.152447 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":87,"threads_started":1}
I20260620 00:20:37.743713 15708 raft_consensus.cc:482] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:37.743813 15708 raft_consensus.cc:504] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:37.744024 15708 leader_election.cc:288] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485), aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915)
W20260620 00:20:37.744470 15291 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
W20260620 00:20:37.744581 15288 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
I20260620 00:20:37.744648 15288 leader_election.cc:302] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d8a755cb574d87a8a0eebd7e3e3dee; no voters: 14b36fcf88134e9f801650fa8758fc2b, aec66acaa4b5442f8226c144a10d3a8f
I20260620 00:20:37.744715 15708 raft_consensus.cc:2726] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:20:37.954021 15709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:37.953800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:53745)
0620 00:20:37.953934 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:37.953973 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260620 00:20:38.005733 15710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:38.005438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:33387)
0620 00:20:38.005629 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:38.005669 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":129,"threads_started":1}
I20260620 00:20:38.153324 15711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:38.153057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:45795)
0620 00:20:38.153216 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:38.153262 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":85,"threads_started":1}
I20260620 00:20:38.954836 15712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:38.954592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:42561)
0620 00:20:38.954745 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:38.954788 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":81,"threads_started":1}
I20260620 00:20:39.006507 15713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:39.006263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:52759)
0620 00:20:39.006423 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:39.006460 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":77,"threads_started":1}
I20260620 00:20:39.154302 15714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:39.153998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:41179)
0620 00:20:39.154171 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:39.154209 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":99,"threads_started":1}
I20260620 00:20:39.259240 15715 raft_consensus.cc:482] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:39.259362 15715 raft_consensus.cc:504] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:39.259624 15715 leader_election.cc:288] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485), aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915)
W20260620 00:20:39.260111 15288 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
W20260620 00:20:39.260217 15291 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
I20260620 00:20:39.260260 15291 leader_election.cc:302] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d8a755cb574d87a8a0eebd7e3e3dee; no voters: 14b36fcf88134e9f801650fa8758fc2b, aec66acaa4b5442f8226c144a10d3a8f
I20260620 00:20:39.260358 15715 raft_consensus.cc:2726] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:20:39.955734 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:39.955496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:45081)
0620 00:20:39.955639 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:39.955687 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":72,"threads_started":1}
I20260620 00:20:40.007424 15717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:40.007163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:45421)
0620 00:20:40.007330 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:40.007375 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":66,"threads_started":1}
I20260620 00:20:40.155087 15718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:40.154854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:44229)
0620 00:20:40.154996 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:40.155039 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260620 00:20:40.956557 15719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:40.956311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:57765)
0620 00:20:40.956457 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:40.956510 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
I20260620 00:20:41.008431 15720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:41.008062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:33125)
0620 00:20:41.008318 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:41.008385 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":191,"threads_started":1}
I20260620 00:20:41.155931 15721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:41.155686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:57009)
0620 00:20:41.155837 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:41.155882 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":65,"threads_started":1}
I20260620 00:20:41.358969 15722 raft_consensus.cc:482] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:41.359105 15722 raft_consensus.cc:504] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "14b36fcf88134e9f801650fa8758fc2b" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 34485 } } peers { permanent_uuid: "aec66acaa4b5442f8226c144a10d3a8f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39915 } } peers { permanent_uuid: "c1d8a755cb574d87a8a0eebd7e3e3dee" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45347 } }
I20260620 00:20:41.359436 15722 leader_election.cc:288] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485), aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915)
W20260620 00:20:41.359889 15291 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 14b36fcf88134e9f801650fa8758fc2b (127.0.98.2:34485): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
W20260620 00:20:41.359968 15288 leader_election.cc:341] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer aec66acaa4b5442f8226c144a10d3a8f (127.0.98.3:39915): Not found: Tablet not found: c19bc36dcfc24cf085736cd426fe8013
I20260620 00:20:41.360038 15288 leader_election.cc:302] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c1d8a755cb574d87a8a0eebd7e3e3dee; no voters: 14b36fcf88134e9f801650fa8758fc2b, aec66acaa4b5442f8226c144a10d3a8f
I20260620 00:20:41.360172 15722 raft_consensus.cc:2726] T c19bc36dcfc24cf085736cd426fe8013 P c1d8a755cb574d87a8a0eebd7e3e3dee [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:20:41.957432 15723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:41.957127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.1:57017)
0620 00:20:41.957329 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:41.957371 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":106,"threads_started":1}
I20260620 00:20:42.009409 15724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:42.009140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.2:52545)
0620 00:20:42.009318 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:42.009361 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":89,"threads_started":1}
I20260620 00:20:42.154932 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 15243
I20260620 00:20:42.156947 15725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:42.156715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33365 (local address 127.0.98.3:60471)
0620 00:20:42.156859 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:42.156905 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:33365: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260620 00:20:42.160173 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 15406
I20260620 00:20:42.164772 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 15539
2026-06-20T00:20:42Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18641 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:20:42 dist-test-slave-r81b krb5kdc[15728](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:20:42 dist-test-slave-r81b krb5kdc[15728](info): set up 2 sockets
Jun 20 00:20:42 dist-test-slave-r81b krb5kdc[15728](info): commencing operation
krb5kdc: starting...
W20260620 00:20:43.212525 392 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:20:43 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:20:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:20:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:20:43.335011 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41889
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:43.416924 15741 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:43.417224 15741 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:43.417273 15741 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:43.418594 15741 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 00:20:43.418680 15741 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 00:20:43.418704 15741 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 00:20:43.418725 15741 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:20:43.418757 15741 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:43.418788 15741 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:43.418818 15741 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:20:43.418849 15741 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:20:43.420579 15741 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15741
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:43.420903 15741 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:43.421156 15741 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:43.426409 15746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:43.426467 15749 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:43.426574 15741 server_base.cc:1034] running on GCE node
W20260620 00:20:43.426409 15747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:43.427035 15741 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:43.427348 15741 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:43.428539 15741 hybrid_clock.cc:648] HybridClock initialized: now 1781914843428535 us; error 48 us; skew 500 ppm
Jun 20 00:20:43 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:43.431252 15741 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:20:43.431672 15741 webserver.cc:460] Webserver started at http://127.0.98.62:33261/ using document root <none> and password file <none>
I20260620 00:20:43.431890 15741 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:43.431936 15741 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:43.432029 15741 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:43.433159 15741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "e7e74115cc814bebba9cc589c55f8bec"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "2234c34dc7b322dc5d01f50f320a560f"
server_key_iv: "a59f9da53af9a3ad8bb6de2195084ee5"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.433468 15741 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "e7e74115cc814bebba9cc589c55f8bec"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "2234c34dc7b322dc5d01f50f320a560f"
server_key_iv: "a59f9da53af9a3ad8bb6de2195084ee5"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.434804 15741 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:20:43.435557 15756 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:43.435738 15741 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:20:43.435829 15741 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "e7e74115cc814bebba9cc589c55f8bec"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "2234c34dc7b322dc5d01f50f320a560f"
server_key_iv: "a59f9da53af9a3ad8bb6de2195084ee5"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.435881 15741 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:43.467123 15741 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:43.469797 15741 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:43.469923 15741 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:43.474088 15741 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41889
I20260620 00:20:43.474423 15808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41889 every 8 connection(s)
I20260620 00:20:43.474507 15741 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:20:43.475304 15809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:43.477695 15809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Bootstrap starting.
I20260620 00:20:43.478322 15809 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:43.478621 15809 log.cc:824] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:43.479423 15809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: No bootstrap required, opened a new log
I20260620 00:20:43.480614 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 15741
I20260620 00:20:43.480695 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:20:43.480898 392 external_mini_cluster.cc:1428] Setting key 081ee967ed9908f6772bdf2518207c25
I20260620 00:20:43.481475 15809 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:20:43.481722 15809 raft_consensus.cc:374] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:43.481768 15809 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e74115cc814bebba9cc589c55f8bec, State: Initialized, Role: FOLLOWER
I20260620 00:20:43.481899 15809 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:20:43.481977 15809 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:20:43.482014 15809 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:20:43.482082 15809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:43.483075 15809 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:20:43.483189 15809 leader_election.cc:302] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7e74115cc814bebba9cc589c55f8bec; no voters:
I20260620 00:20:43.483350 15809 leader_election.cc:288] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:20:43.483440 15814 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:20:43.483609 15809 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:20:43.483623 15814 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 1 LEADER]: Becoming Leader. State: Replica: e7e74115cc814bebba9cc589c55f8bec, State: Running, Role: LEADER
I20260620 00:20:43.483937 15814 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:20:43.484405 15815 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e7e74115cc814bebba9cc589c55f8bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } } }
I20260620 00:20:43.484485 15815 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:43.484845 15816 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7e74115cc814bebba9cc589c55f8bec. Latest consensus state: current_term: 1 leader_uuid: "e7e74115cc814bebba9cc589c55f8bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } } }
I20260620 00:20:43.484921 15816 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: This master's current role is: LEADER
I20260620 00:20:43.485322 15823 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:20:43.485987 15823 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:20:43.488113 15823 catalog_manager.cc:1347] Generated new cluster ID: 69dcc52e280744e982fcceebe960f272
I20260620 00:20:43.488199 15823 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:20:43 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:43.492662 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:43.481483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38748 (local address 127.0.98.62:41889)
0620 00:20:43.481695 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:43.481705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:43.481742 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:20:43.481866 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:43.481873 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:43.481931 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:43.482089 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:43.482876 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.484403 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.485611 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.486248 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.486383 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:43.489126 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:43.489131 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:43.489131 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:43.489145 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:43.490977 (+ 1832us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:43.491371 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:43.491374 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:43.491375 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:43.491489 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:43.491718 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:43.491719 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:43.491719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:43.491980 (+ 261us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:43.492144 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:43.492288 (+ 144us) server_negotiation.cc:299] Negotiation successful
0620 00:20:43.492441 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":102,"threads_started":1}
I20260620 00:20:43.503136 15823 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:20:43.503770 15823 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:20:43.507668 15823 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Generated new TSK 0
I20260620 00:20:43.507881 15823 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:20:43.543382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41889
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:43.628508 15835 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:43.628738 15835 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:43.628765 15835 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:43.630061 15835 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:43.630116 15835 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:43.630146 15835 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:20:43.631706 15835 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15835
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:43.631990 15835 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:43.632300 15835 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:43.637669 15843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:43.637669 15840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:43.637821 15835 server_base.cc:1034] running on GCE node
W20260620 00:20:43.637693 15841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:43.638236 15835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:43.638568 15835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:43.639721 15835 hybrid_clock.cc:648] HybridClock initialized: now 1781914843639699 us; error 37 us; skew 500 ppm
Jun 20 00:20:43 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:43.642586 15835 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:20:43.643142 15835 webserver.cc:460] Webserver started at http://127.0.98.1:35763/ using document root <none> and password file <none>
I20260620 00:20:43.643388 15835 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:43.643461 15835 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:43.643615 15835 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:43.644805 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "f5ad9831734b4046b5dc4b177b6fbe7f"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "58cd42becbbe9cc1832dc3829e342e96"
server_key_iv: "8bd624fea58e3294b2780c34bc0537bd"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.645191 15835 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f5ad9831734b4046b5dc4b177b6fbe7f"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "58cd42becbbe9cc1832dc3829e342e96"
server_key_iv: "8bd624fea58e3294b2780c34bc0537bd"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.646669 15835 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:20:43.647568 15850 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:43.647761 15835 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:43.647868 15835 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "f5ad9831734b4046b5dc4b177b6fbe7f"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "58cd42becbbe9cc1832dc3829e342e96"
server_key_iv: "8bd624fea58e3294b2780c34bc0537bd"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.647980 15835 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:43.670488 15835 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:43.673098 15835 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:43.673327 15835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:43.673601 15835 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:43.673956 15835 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:43.674016 15835 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:43.674058 15835 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:43.674104 15835 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:43.680330 15835 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42487
I20260620 00:20:43.680423 15963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42487 every 8 connection(s)
I20260620 00:20:43.680783 15835 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:20:43 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:43.689066 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 15835
I20260620 00:20:43.689195 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:20:43.689420 392 external_mini_cluster.cc:1428] Setting key 72e76894e194b6eba907e9a8b41e04bc
I20260620 00:20:43.692541 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:43.681705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48523 (local address 127.0.98.62:41889)
0620 00:20:43.681796 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:43.681798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:43.682062 (+ 264us) server_negotiation.cc:407] Connection header received
0620 00:20:43.682632 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:43.682633 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:43.682651 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:43.682694 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:43.683872 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.684545 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.685792 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.686002 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.686095 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:43.688365 (+ 2270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:43.688366 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:43.688367 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:43.688384 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:43.690390 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:43.690990 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:43.690991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:43.690992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:43.691069 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:43.691532 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:43.691534 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:43.691534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:43.691675 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:43.691820 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:43.692238 (+ 418us) server_negotiation.cc:299] Negotiation successful
0620 00:20:43.692373 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:20:43.692579 15966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:43.681648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:48523)
0620 00:20:43.681909 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:43.681948 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:43.682493 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:43.682791 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:43.682794 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:43.683455 (+ 661us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:43.683761 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:43.683768 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.684648 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.684649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:43.685692 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:43.685694 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.686109 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.686110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:43.686246 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:43.686787 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:43.686795 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:43.688260 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:43.690473 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:43.690481 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:43.690484 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:43.690869 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:43.691237 (+ 368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:43.691238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:43.691239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:43.691415 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:43.691797 (+ 382us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:43.691806 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:43.692014 (+ 208us) client_negotiation.cc:769] Sending connection context
0620 00:20:43.692193 (+ 179us) client_negotiation.cc:240] Negotiation successful
0620 00:20:43.692369 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":89,"threads_started":1}
I20260620 00:20:43.693166 15964 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41889
I20260620 00:20:43.693269 15964 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:43.693480 15964 heartbeater.cc:511] Master 127.0.98.62:41889 requested a full tablet report, sending...
I20260620 00:20:43.694193 15773 ts_manager.cc:194] Registered new tserver with Master: f5ad9831734b4046b5dc4b177b6fbe7f (127.0.98.1:42487)
I20260620 00:20:43.694923 15773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:48523
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:20:43.718508 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41889
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:43.802778 15969 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:43.803063 15969 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:43.803108 15969 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:43.804316 15969 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:43.804399 15969 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:43.804445 15969 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:20:43.806026 15969 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15969
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:43.806330 15969 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:43.806588 15969 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:43.812301 15977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:43.812310 15975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:43.812321 15969 server_base.cc:1034] running on GCE node
W20260620 00:20:43.812441 15974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:43.812786 15969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:43.813038 15969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:43.814213 15969 hybrid_clock.cc:648] HybridClock initialized: now 1781914843814155 us; error 65 us; skew 500 ppm
Jun 20 00:20:43 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:43.817016 15969 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:20:43.817567 15969 webserver.cc:460] Webserver started at http://127.0.98.2:35329/ using document root <none> and password file <none>
I20260620 00:20:43.817818 15969 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:43.817889 15969 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:43.818046 15969 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:43.819150 15969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "700b63c333a74dae99c9a43b2043170e"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "685942f0f6c8aa3dda511f66f9d5733b"
server_key_iv: "4b9b9b244b26e50527cef61d71401540"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.819527 15969 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "700b63c333a74dae99c9a43b2043170e"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "685942f0f6c8aa3dda511f66f9d5733b"
server_key_iv: "4b9b9b244b26e50527cef61d71401540"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.820964 15969 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:20:43.821810 15984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:43.822019 15969 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:43.822122 15969 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "700b63c333a74dae99c9a43b2043170e"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "685942f0f6c8aa3dda511f66f9d5733b"
server_key_iv: "4b9b9b244b26e50527cef61d71401540"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.822237 15969 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:43.845526 15969 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:43.848004 15969 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:43.848129 15969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:43.848416 15969 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:43.848768 15969 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:43.848814 15969 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:43.848836 15969 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:43.848879 15969 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:43.855126 15969 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40979
I20260620 00:20:43.855163 16097 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40979 every 8 connection(s)
I20260620 00:20:43.855504 15969 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:20:43 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:43.864504 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 15969
I20260620 00:20:43.864655 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:20:43.864908 392 external_mini_cluster.cc:1428] Setting key 427368dadce28017f07b354cd3ff5911
I20260620 00:20:43.866133 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:43.856290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:33925)
0620 00:20:43.856462 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:43.856505 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:43.857081 (+ 576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:43.857392 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:43.857396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:43.858041 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:43.858257 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:43.858263 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.859031 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.859033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:43.859956 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:43.859958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.860474 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.860475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:43.860630 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:43.861184 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:43.861193 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:43.862679 (+ 1486us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:43.864403 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:43.864410 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:43.864412 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:43.864753 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:43.865038 (+ 285us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:43.865039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:43.865039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:43.865198 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:43.865507 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:43.865514 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:43.865697 (+ 183us) client_negotiation.cc:769] Sending connection context
0620 00:20:43.865817 (+ 120us) client_negotiation.cc:240] Negotiation successful
0620 00:20:43.865957 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
I20260620 00:20:43.866135 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:43.856281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33925 (local address 127.0.98.62:41889)
0620 00:20:43.856375 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:43.856378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:43.856620 (+ 242us) server_negotiation.cc:407] Connection header received
0620 00:20:43.857222 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:43.857224 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:43.857237 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:43.857292 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:43.858368 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.858928 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.860062 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:43.860365 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:43.860461 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:43.862794 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:43.862796 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:43.862797 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:43.862805 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:43.864268 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:43.864863 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:43.864865 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:43.864865 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:43.864936 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:43.865269 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:43.865270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:43.865271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:43.865410 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:43.865470 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:43.865870 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 00:20:43.865975 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 00:20:43.866722 16098 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41889
I20260620 00:20:43.866839 16098 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:43.867066 16098 heartbeater.cc:511] Master 127.0.98.62:41889 requested a full tablet report, sending...
I20260620 00:20:43.867517 15773 ts_manager.cc:194] Registered new tserver with Master: 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
I20260620 00:20:43.867934 15773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:33925
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:20:43.891402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41889
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:20:43.973963 16103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:20:43.974215 16103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:20:43.974254 16103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:20:43.975459 16103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:20:43.975533 16103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:20:43.975592 16103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:20:43.977257 16103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16103
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:20:43.977581 16103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:20:43.977860 16103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:20:43.983243 16109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:20:43.983253 16108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:43.983283 16103 server_base.cc:1034] running on GCE node
W20260620 00:20:43.983273 16111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:20:43.983845 16103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:20:43.984126 16103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:20:43.985355 16103 hybrid_clock.cc:648] HybridClock initialized: now 1781914843985289 us; error 94 us; skew 500 ppm
Jun 20 00:20:43 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:20:43.988219 16103 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:20:43.988675 16103 webserver.cc:460] Webserver started at http://127.0.98.3:39739/ using document root <none> and password file <none>
I20260620 00:20:43.988938 16103 fs_manager.cc:359] Metadata directory not provided
I20260620 00:20:43.989009 16103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:20:43.989171 16103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:20:43.990285 16103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "cf6b2becce9a402cb864a7a3c41bf46b"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "08dcf72954432d6a0ee2709d826b57c9"
server_key_iv: "ea2ef94c403db71f6376b88d010f3c5b"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.990667 16103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "cf6b2becce9a402cb864a7a3c41bf46b"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "08dcf72954432d6a0ee2709d826b57c9"
server_key_iv: "ea2ef94c403db71f6376b88d010f3c5b"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.992231 16103 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:20:43.993227 16118 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:43.993419 16103 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:20:43.993520 16103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "cf6b2becce9a402cb864a7a3c41bf46b"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "08dcf72954432d6a0ee2709d826b57c9"
server_key_iv: "ea2ef94c403db71f6376b88d010f3c5b"
server_key_version: "encryptionkey@0"
I20260620 00:20:43.993623 16103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:20:44.007987 16103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:20:44.010427 16103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:20:44.010638 16103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:20:44.010892 16103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:20:44.011206 16103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:20:44.011261 16103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:44.011324 16103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:20:44.011371 16103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:20:44.017488 16103 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45209
I20260620 00:20:44.017527 16231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45209 every 8 connection(s)
I20260620 00:20:44.017897 16103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:20:44 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914843, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:20:44.026826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 16103
I20260620 00:20:44.026945 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:20:44.027136 392 external_mini_cluster.cc:1428] Setting key 22f6dd037e69074024c85ab7a8417de3
I20260620 00:20:44.028354 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.018668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55005 (local address 127.0.98.62:41889)
0620 00:20:44.018715 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:44.018716 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:44.018946 (+ 230us) server_negotiation.cc:407] Connection header received
0620 00:20:44.019502 (+ 556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:44.019503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:44.019515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:44.019564 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:44.020621 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.021158 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.022175 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.022369 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.022475 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.025081 (+ 2606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:44.025082 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:44.025084 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:44.025097 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:44.026731 (+ 1634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:44.027295 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:44.027297 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:44.027298 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:44.027361 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:44.027638 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:44.027639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:44.027639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:44.027759 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:44.027833 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:44.028167 (+ 334us) server_negotiation.cc:299] Negotiation successful
0620 00:20:44.028261 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:20:44.028558 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.018640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:55005)
0620 00:20:44.018815 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:44.018851 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:44.019391 (+ 540us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:44.019641 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:44.019646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:44.020292 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:20:44.020518 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.020523 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.021257 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.021258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.022079 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.022081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.022488 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.022489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.022614 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.023245 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:20:44.023254 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:20:44.024893 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:20:44.026833 (+ 1940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:44.026842 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:44.026844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:44.027220 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:44.027434 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:20:44.027435 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:20:44.027435 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:20:44.027568 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:20:44.027833 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:20:44.027840 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:20:44.027991 (+ 151us) client_negotiation.cc:769] Sending connection context
0620 00:20:44.028112 (+ 121us) client_negotiation.cc:240] Negotiation successful
0620 00:20:44.028308 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":80,"threads_started":1}
I20260620 00:20:44.029057 16232 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41889
I20260620 00:20:44.029157 16232 heartbeater.cc:464] Registering TS with master...
I20260620 00:20:44.029304 16232 heartbeater.cc:511] Master 127.0.98.62:41889 requested a full tablet report, sending...
I20260620 00:20:44.029685 15773 ts_manager.cc:194] Registered new tserver with Master: cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209)
I20260620 00:20:44.030182 15773 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55005
I20260620 00:20:44.039023 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:20:44.046847 15813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.040561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38760 (local address 127.0.98.62:41889)
0620 00:20:44.040611 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:44.040613 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:44.040670 (+ 57us) server_negotiation.cc:407] Connection header received
0620 00:20:44.040729 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:44.040729 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:44.040738 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:44.040793 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:20:44.041387 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.041875 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.042609 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.042799 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.042863 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.044076 (+ 1213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:20:44.044077 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:20:44.044077 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:20:44.044084 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:20:44.045655 (+ 1571us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:44.046055 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:44.046056 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:44.046057 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:44.046116 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:20:44.046312 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:20:44.046313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:20:44.046313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:20:44.046463 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:20:44.046543 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:44.046707 (+ 164us) server_negotiation.cc:299] Negotiation successful
0620 00:20:44.046772 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:20:44.048883 15773 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:20:44.050632 15773 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:20:44.058045 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.053559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49652 (local address 127.0.98.3:45209)
0620 00:20:44.053776 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:44.053783 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:44.053794 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:20:44.053837 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:44.053840 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:44.053856 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:44.053931 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:44.054080 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.055251 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.056915 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.057607 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.057673 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.057726 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:44.057908 (+ 182us) server_negotiation.cc:299] Negotiation successful
0620 00:20:44.057952 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":99,"threads_started":1}
I20260620 00:20:44.058061 16243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.053387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45209 (local address 127.0.0.1:49652)
0620 00:20:44.053497 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:44.053541 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:44.053612 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:44.053936 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:44.053940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:44.053943 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:44.054021 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.054023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.055356 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.055357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.056821 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.056824 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.057687 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.057689 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.057797 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.057801 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:20:44.057884 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 00:20:44.057960 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":45,"thread_start_us":64,"threads_started":1}
I20260620 00:20:44.058061 16245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.053404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40979 (local address 127.0.0.1:48562)
0620 00:20:44.053850 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:44.053860 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:44.053922 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:44.054338 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:44.054339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:44.054341 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:44.054381 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.054383 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.055588 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.055589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.056553 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.056555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.057689 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.057690 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.057781 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.057800 (+ 19us) client_negotiation.cc:769] Sending connection context
0620 00:20:44.057884 (+ 84us) client_negotiation.cc:240] Negotiation successful
0620 00:20:44.057960 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"mutex_wait_us":29,"thread_start_us":113,"threads_started":1}
I20260620 00:20:44.058310 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.053667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48562 (local address 127.0.98.2:40979)
0620 00:20:44.053834 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:44.053837 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:44.054119 (+ 282us) server_negotiation.cc:407] Connection header received
0620 00:20:44.054163 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:44.054165 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:44.054181 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:44.054254 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:44.054457 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.055489 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.056750 (+ 1261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.057564 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.057652 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.057720 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:44.058190 (+ 470us) server_negotiation.cc:299] Negotiation successful
0620 00:20:44.058235 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":88,"threads_started":1}
I20260620 00:20:44.058938 16166 tablet_service.cc:1467] Processing CreateTablet for tablet 008c15791fee4c5299c6d061bc8eeddc (DEFAULT_TABLE table=test-table [id=66b91ae8d7d7498ea26aea19fab21490]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:44.059108 16032 tablet_service.cc:1467] Processing CreateTablet for tablet 008c15791fee4c5299c6d061bc8eeddc (DEFAULT_TABLE table=test-table [id=66b91ae8d7d7498ea26aea19fab21490]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:44.059321 16166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 008c15791fee4c5299c6d061bc8eeddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:44.059365 16032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 008c15791fee4c5299c6d061bc8eeddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:44.062208 16250 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Bootstrap starting.
I20260620 00:20:44.062871 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.053404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42487 (local address 127.0.0.1:38678)
0620 00:20:44.053848 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:44.053860 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:44.053918 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:44.054296 (+ 378us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:44.054297 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:44.054299 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:44.054349 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.054352 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.055241 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.055242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.056823 (+ 1581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.056825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.062654 (+ 5829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.062655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.062717 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.062722 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:20:44.062767 (+ 45us) client_negotiation.cc:240] Negotiation successful
0620 00:20:44.062808 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":87,"spinlock_wait_cycles":384,"thread_start_us":104,"threads_started":1}
I20260620 00:20:44.062925 16250 tablet_bootstrap.cc:654] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:44.062953 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.053699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38678 (local address 127.0.98.1:42487)
0620 00:20:44.054037 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:44.054041 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:44.054050 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:20:44.054122 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:44.054124 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:44.054138 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:44.054211 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:44.054427 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.055129 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.057765 (+ 2636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.062553 (+ 4788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.062646 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.062711 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:44.062793 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 00:20:44.062849 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":304,"thread_start_us":73,"threads_started":1}
I20260620 00:20:44.063498 16250 log.cc:824] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:44.063877 16249 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: Bootstrap starting.
I20260620 00:20:44.063841 15898 tablet_service.cc:1467] Processing CreateTablet for tablet 008c15791fee4c5299c6d061bc8eeddc (DEFAULT_TABLE table=test-table [id=66b91ae8d7d7498ea26aea19fab21490]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:20:44.064247 15898 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 008c15791fee4c5299c6d061bc8eeddc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:20:44.064378 16250 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: No bootstrap required, opened a new log
I20260620 00:20:44.064456 16250 ts_tablet_manager.cc:1397] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:20:44.064661 16249 tablet_bootstrap.cc:654] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:44.065266 16249 log.cc:824] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:44.066115 16249 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: No bootstrap required, opened a new log
I20260620 00:20:44.066205 16249 ts_tablet_manager.cc:1397] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:20:44.066555 16250 raft_consensus.cc:348] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:20:44.066754 16250 raft_consensus.cc:374] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:44.066788 16250 raft_consensus.cc:729] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700b63c333a74dae99c9a43b2043170e, State: Initialized, Role: FOLLOWER
I20260620 00:20:44.066910 16250 consensus_queue.cc:260] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:20:44.067211 16250 ts_tablet_manager.cc:1428] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:20:44.067240 16098 heartbeater.cc:503] Master 127.0.98.62:41889 was elected leader, sending a full tablet report...
I20260620 00:20:44.067438 16254 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Bootstrap starting.
I20260620 00:20:44.068199 16254 tablet_bootstrap.cc:654] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Neither blocks nor log segments found. Creating new log.
I20260620 00:20:44.068440 16249 raft_consensus.cc:348] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:20:44.068653 16249 raft_consensus.cc:374] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:44.068693 16249 raft_consensus.cc:729] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6b2becce9a402cb864a7a3c41bf46b, State: Initialized, Role: FOLLOWER
I20260620 00:20:44.068840 16249 consensus_queue.cc:260] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:20:44.068874 16254 log.cc:824] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:20:44.069182 16249 ts_tablet_manager.cc:1428] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:20:44.069298 16232 heartbeater.cc:503] Master 127.0.98.62:41889 was elected leader, sending a full tablet report...
I20260620 00:20:44.069895 16254 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: No bootstrap required, opened a new log
I20260620 00:20:44.069983 16254 ts_tablet_manager.cc:1397] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260620 00:20:44.071483 16254 raft_consensus.cc:348] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:20:44.071640 16254 raft_consensus.cc:374] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:20:44.071666 16254 raft_consensus.cc:729] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ad9831734b4046b5dc4b177b6fbe7f, State: Initialized, Role: FOLLOWER
I20260620 00:20:44.071782 16254 consensus_queue.cc:260] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:20:44.072028 16254 ts_tablet_manager.cc:1428] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:20:44.072090 15964 heartbeater.cc:503] Master 127.0.98.62:41889 was elected leader, sending a full tablet report...
W20260620 00:20:44.106045 16099 tablet.cc:2369] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:20:44.181566 15965 tablet.cc:2369] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:20:44.220050 16255 raft_consensus.cc:482] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:20:44.220228 16255 raft_consensus.cc:504] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:20:44.220731 16255 leader_election.cc:288] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f5ad9831734b4046b5dc4b177b6fbe7f (127.0.98.1:42487), 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
I20260620 00:20:44.223985 16258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.220976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42487 (local address 127.0.98.3:46671)
0620 00:20:44.221215 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:44.221225 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:44.221288 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:44.221452 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:44.221453 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:44.221455 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:44.221565 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.221567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.222277 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.222277 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.223141 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.223144 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.223723 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.223724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.223780 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.223783 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:20:44.223850 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 00:20:44.223893 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"mutex_wait_us":73,"thread_start_us":93,"threads_started":1}
I20260620 00:20:44.223970 16248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.221068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46671 (local address 127.0.98.1:42487)
0620 00:20:44.221122 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:44.221124 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:44.221285 (+ 161us) server_negotiation.cc:407] Connection header received
0620 00:20:44.221360 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:44.221361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:44.221374 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:44.221439 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:44.221649 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.222187 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.223252 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.223618 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.223665 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.223675 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:44.223858 (+ 183us) server_negotiation.cc:299] Negotiation successful
0620 00:20:44.223901 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:20:44.224089 16247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.221114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47929 (local address 127.0.98.2:40979)
0620 00:20:44.221182 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:44.221184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:44.221196 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:20:44.221265 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:44.221266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:44.221282 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:44.221341 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:20:44.221533 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.222098 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.223231 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.223673 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.223784 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.223796 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:44.223974 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 00:20:44.224024 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:20:44.224072 16234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.220976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40979 (local address 127.0.98.3:47929)
0620 00:20:44.221063 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:44.221079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:20:44.221193 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:20:44.221356 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:20:44.221357 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:20:44.221360 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:20:44.221447 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.221449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.222184 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.222185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.223120 (+ 935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:20:44.223122 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.223786 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.223787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:20:44.223879 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.223882 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:20:44.223952 (+ 70us) client_negotiation.cc:240] Negotiation successful
0620 00:20:44.223998 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":61}
I20260620 00:20:44.224640 15918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "008c15791fee4c5299c6d061bc8eeddc" candidate_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" is_pre_election: true
I20260620 00:20:44.224686 16052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "008c15791fee4c5299c6d061bc8eeddc" candidate_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700b63c333a74dae99c9a43b2043170e" is_pre_election: true
I20260620 00:20:44.224821 15918 raft_consensus.cc:2454] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6b2becce9a402cb864a7a3c41bf46b in term 0.
I20260620 00:20:44.224843 16052 raft_consensus.cc:2454] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cf6b2becce9a402cb864a7a3c41bf46b in term 0.
I20260620 00:20:44.225069 16122 leader_election.cc:302] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6b2becce9a402cb864a7a3c41bf46b, f5ad9831734b4046b5dc4b177b6fbe7f; no voters:
I20260620 00:20:44.225220 16255 raft_consensus.cc:2781] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:20:44.225273 16255 raft_consensus.cc:482] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:20:44.225291 16255 raft_consensus.cc:3037] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:44.226094 16255 raft_consensus.cc:504] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:20:44.226289 16255 leader_election.cc:288] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [CANDIDATE]: Term 1 election: Requested vote from peers f5ad9831734b4046b5dc4b177b6fbe7f (127.0.98.1:42487), 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
I20260620 00:20:44.226462 15918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "008c15791fee4c5299c6d061bc8eeddc" candidate_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f"
I20260620 00:20:44.226549 15918 raft_consensus.cc:3037] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:44.226521 16052 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "008c15791fee4c5299c6d061bc8eeddc" candidate_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700b63c333a74dae99c9a43b2043170e"
I20260620 00:20:44.226632 16052 raft_consensus.cc:3037] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:20:44.227360 15918 raft_consensus.cc:2454] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6b2becce9a402cb864a7a3c41bf46b in term 1.
I20260620 00:20:44.227536 16122 leader_election.cc:302] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6b2becce9a402cb864a7a3c41bf46b, f5ad9831734b4046b5dc4b177b6fbe7f; no voters:
I20260620 00:20:44.227553 16052 raft_consensus.cc:2454] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cf6b2becce9a402cb864a7a3c41bf46b in term 1.
I20260620 00:20:44.227644 16255 raft_consensus.cc:2781] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:20:44.227797 16255 raft_consensus.cc:686] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 1 LEADER]: Becoming Leader. State: Replica: cf6b2becce9a402cb864a7a3c41bf46b, State: Running, Role: LEADER
I20260620 00:20:44.227900 16255 consensus_queue.cc:237] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:20:44.228825 15772 catalog_manager.cc:5495] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b reported cstate change: term changed from 0 to 1, leader changed from <none> to cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } health_report { overall_health: UNKNOWN } } }
I20260620 00:20:44.241098 16246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.238247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.0.98.3:45209)
0620 00:20:44.238304 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:20:44.238306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:20:44.238316 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:20:44.238381 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:20:44.238382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:20:44.238402 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:20:44.238462 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:20:44.238614 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.239184 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.240037 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:20:44.240350 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:20:44.240409 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:20:44.240643 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:20:44.240752 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:20:44.240835 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:20:44.240947 (+ 112us) server_negotiation.cc:299] Negotiation successful
0620 00:20:44.240972 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260620 00:20:44.243080 15918 raft_consensus.cc:1261] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Refusing update from remote peer cf6b2becce9a402cb864a7a3c41bf46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:44.243080 16052 raft_consensus.cc:1261] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Refusing update from remote peer cf6b2becce9a402cb864a7a3c41bf46b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:20:44.243453 16255 consensus_queue.cc:1035] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:44.243594 16259 consensus_queue.cc:1035] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:20:44.245940 16262 mvcc.cc:205] Tried to move back new op lower bound from 7298723202016485376 to 7298723201957974016. Current Snapshot: MvccSnapshot[applied={T|T < 7298723202016485376}]
I20260620 00:20:44.246943 16263 mvcc.cc:205] Tried to move back new op lower bound from 7298723202016485376 to 7298723201957974016. Current Snapshot: MvccSnapshot[applied={T|T < 7298723202016485376 or (T in {7298723202016485376})}]
I20260620 00:20:44.248476 15772 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:20:44.248587 15772 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:20:44.249363 15772 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Sending DeleteTablet for 3 replicas of tablet 008c15791fee4c5299c6d061bc8eeddc
I20260620 00:20:44.249714 15898 tablet_service.cc:1514] Processing DeleteTablet for tablet 008c15791fee4c5299c6d061bc8eeddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38678
I20260620 00:20:44.249742 16166 tablet_service.cc:1514] Processing DeleteTablet for tablet 008c15791fee4c5299c6d061bc8eeddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49652
I20260620 00:20:44.249893 16032 tablet_service.cc:1514] Processing DeleteTablet for tablet 008c15791fee4c5299c6d061bc8eeddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:48562
I20260620 00:20:44.249962 16273 tablet_replica.cc:330] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: stopping tablet replica
I20260620 00:20:44.250078 16273 raft_consensus.cc:2229] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:20:44.250183 16273 raft_consensus.cc:2258] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:20:44.250221 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 15835
I20260620 00:20:44.251631 16273 ts_tablet_manager.cc:1905] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:20:44.253008 16274 tablet_replica.cc:330] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: stopping tablet replica
I20260620 00:20:44.253160 16274 raft_consensus.cc:2229] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:20:44.253304 16274 raft_consensus.cc:2258] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:20:44.253665 16273 ts_tablet_manager.cc:1918] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:20:44.253744 16273 log.cc:1196] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/008c15791fee4c5299c6d061bc8eeddc
I20260620 00:20:44.254069 16273 ts_tablet_manager.cc:1939] T 008c15791fee4c5299c6d061bc8eeddc P cf6b2becce9a402cb864a7a3c41bf46b: Deleting consensus metadata
I20260620 00:20:44.254561 15759 catalog_manager.cc:4841] TS cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209): tablet 008c15791fee4c5299c6d061bc8eeddc (table test-table [id=66b91ae8d7d7498ea26aea19fab21490]) successfully deleted
W20260620 00:20:44.257014 15760 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42487 (error 108)
I20260620 00:20:44.257189 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 15969
I20260620 00:20:44.258947 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.258770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42487 (local address 127.0.0.1:38686)
0620 00:20:44.258846 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:44.258899 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260620 00:20:44.259093 15760 catalog_manager.cc:4584] TS f5ad9831734b4046b5dc4b177b6fbe7f (127.0.98.1:42487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 008c15791fee4c5299c6d061bc8eeddc: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42487: connect: Connection refused (error 111)
I20260620 00:20:44.262449 16244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:20:44.262306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40979 (local address 127.0.0.1:48568)
0620 00:20:44.262368 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 00:20:44.262408 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:40979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:20:44.262454 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 16103
I20260620 00:20:44.268226 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 15741
I20260620 00:21:04.274178 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41889
--webserver_interface=127.0.98.62
--webserver_port=33261
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:04.358875 16276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:04.359117 16276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:04.359143 16276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:04.360575 16276 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 00:21:04.360641 16276 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 00:21:04.360656 16276 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 00:21:04.360669 16276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:04.360683 16276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:04.360697 16276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:04.360709 16276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:04.360762 16276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:04.362582 16276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:04.362905 16276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:04.363191 16276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:04.373764 16283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:04.373764 16285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:04.373884 16282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:04.374186 16276 server_base.cc:1034] running on GCE node
I20260620 00:21:04.374493 16276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:04.374878 16276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:04.376048 16276 hybrid_clock.cc:648] HybridClock initialized: now 1781914864376036 us; error 29 us; skew 500 ppm
Jun 20 00:21:04 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:04.383797 16276 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:21:04.384441 16276 webserver.cc:460] Webserver started at http://127.0.98.62:33261/ using document root <none> and password file <none>
I20260620 00:21:04.384745 16276 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:04.384814 16276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:04.386565 16276 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:21:04.387418 16292 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:04.387734 16276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:04.387836 16276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "e7e74115cc814bebba9cc589c55f8bec"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "2234c34dc7b322dc5d01f50f320a560f"
server_key_iv: "a59f9da53af9a3ad8bb6de2195084ee5"
server_key_version: "encryptionkey@0"
I20260620 00:21:04.388360 16276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:04.395920 16276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:04.398635 16276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:04.398782 16276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:04.403190 16276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41889
I20260620 00:21:04.403393 16344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41889 every 8 connection(s)
I20260620 00:21:04.403697 16276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:04.405457 16345 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:21:04.406167 16345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Bootstrap starting.
I20260620 00:21:04.410125 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 16276
I20260620 00:21:04.410514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42487
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41889
--webserver_port=35763
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:21:04.415499 16345 log.cc:824] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:04.417591 16345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:21:04.417986 16345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Bootstrap complete.
I20260620 00:21:04.420463 16345 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:21:04.420702 16345 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e74115cc814bebba9cc589c55f8bec, State: Initialized, Role: FOLLOWER
I20260620 00:21:04.420897 16345 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:21:04.420964 16345 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:04.421021 16345 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:04.421072 16345 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 1 FOLLOWER]: Advancing to term 2
I20260620 00:21:04.422200 16345 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:21:04.422324 16345 leader_election.cc:302] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7e74115cc814bebba9cc589c55f8bec; no voters:
I20260620 00:21:04.422569 16345 leader_election.cc:288] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [CANDIDATE]: Term 2 election: Requested vote from peers
I20260620 00:21:04.422760 16349 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 2 FOLLOWER]: Leader election won for term 2
I20260620 00:21:04.422933 16349 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 2 LEADER]: Becoming Leader. State: Replica: e7e74115cc814bebba9cc589c55f8bec, State: Running, Role: LEADER
I20260620 00:21:04.423135 16345 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:04.423137 16349 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:21:04.423662 16350 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e7e74115cc814bebba9cc589c55f8bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } } }
I20260620 00:21:04.423766 16350 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:04.424223 16351 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7e74115cc814bebba9cc589c55f8bec. Latest consensus state: current_term: 2 leader_uuid: "e7e74115cc814bebba9cc589c55f8bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } } }
I20260620 00:21:04.424304 16351 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:04.424420 16356 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:04.425243 16356 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:04.425343 16356 catalog_manager.cc:1259] Loaded cluster ID: 69dcc52e280744e982fcceebe960f272
I20260620 00:21:04.425412 16356 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:04.426829 16356 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:04.427042 16356 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Loaded TSK: 0
I20260620 00:21:04.442420 16356 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Generated new TSK 1
I20260620 00:21:04.442559 16356 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:21:04.504087 16347 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:04.504437 16347 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:04.504482 16347 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:04.505793 16347 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:04.505885 16347 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:04.505935 16347 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:21:04.507608 16347 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35763
--tserver_master_addrs=127.0.98.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16347
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:04.507956 16347 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:04.508273 16347 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:04.513934 16375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:04.513947 16372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:04.513963 16373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:04.514421 16347 server_base.cc:1034] running on GCE node
I20260620 00:21:04.514658 16347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:04.514971 16347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:04.516235 16347 hybrid_clock.cc:648] HybridClock initialized: now 1781914864516200 us; error 45 us; skew 500 ppm
Jun 20 00:21:04 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:04.519315 16347 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:21:04.519833 16347 webserver.cc:460] Webserver started at http://127.0.98.1:35763/ using document root <none> and password file <none>
I20260620 00:21:04.520083 16347 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:04.520195 16347 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:04.521785 16347 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:04.522668 16382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:04.522974 16347 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:04.523144 16347 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "f5ad9831734b4046b5dc4b177b6fbe7f"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "58cd42becbbe9cc1832dc3829e342e96"
server_key_iv: "8bd624fea58e3294b2780c34bc0537bd"
server_key_version: "encryptionkey@0"
I20260620 00:21:04.523638 16347 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:04.559778 16347 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:04.562332 16347 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:04.562456 16347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:04.562772 16347 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:04.563302 16389 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:21:04.565414 16347 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:21:04.565529 16347 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:21:04.565574 16347 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:21:04.566258 16347 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:21:04.566331 16347 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:21:04.566453 16389 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Bootstrap starting.
I20260620 00:21:04.573553 16347 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42487
I20260620 00:21:04.573673 16496 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42487 every 8 connection(s)
I20260620 00:21:04.574009 16347 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:21:04.576399 16389 log.cc:824] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:04.577075 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 16347
I20260620 00:21:04.577410 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40979
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41889
--webserver_port=35329
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:21:04.579214 16389 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:21:04.579526 16389 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Bootstrap complete.
I20260620 00:21:04.580520 16389 ts_tablet_manager.cc:1397] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.009s
Jun 20 00:21:04 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:04.582245 16389 raft_consensus.cc:348] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:21:04.582777 16389 raft_consensus.cc:729] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ad9831734b4046b5dc4b177b6fbe7f, State: Initialized, Role: FOLLOWER
I20260620 00:21:04.582976 16389 consensus_queue.cc:260] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:21:04.583289 16389 ts_tablet_manager.cc:1428] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260620 00:21:04.585933 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.574805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:52885)
0620 00:21:04.575033 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:04.575086 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:04.575608 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:04.576079 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:04.576083 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:04.576437 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:04.576679 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.576685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.578044 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.578045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.578995 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.578999 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.579979 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.579980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.580087 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.580734 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:04.580742 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:04.582155 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:04.584162 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.584168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.584171 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.584412 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.584706 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.584706 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.584707 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.584879 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.585272 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:04.585279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:04.585419 (+ 140us) client_negotiation.cc:769] Sending connection context
0620 00:21:04.585526 (+ 107us) client_negotiation.cc:240] Negotiation successful
0620 00:21:04.585708 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":132,"threads_started":1}
I20260620 00:21:04.585935 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.574947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52885 (local address 127.0.98.62:41889)
0620 00:21:04.575149 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.575156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.575195 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:21:04.575854 (+ 659us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.575859 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.575905 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.576055 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:04.576801 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.577912 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.579109 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.579762 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.579866 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.582272 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:04.582276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:04.582277 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:04.582293 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:04.584024 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.584520 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.584524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.584524 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.584622 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.584956 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.584957 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.584958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.585195 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:04.585293 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.585594 (+ 301us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.585726 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260620 00:21:04.586865 16497 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41889
I20260620 00:21:04.586977 16497 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:04.587190 16497 heartbeater.cc:511] Master 127.0.98.62:41889 requested a full tablet report, sending...
I20260620 00:21:04.588032 16309 ts_manager.cc:194] Registered new tserver with Master: f5ad9831734b4046b5dc4b177b6fbe7f (127.0.98.1:42487)
I20260620 00:21:04.589457 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52885
I20260620 00:21:04.590544 16497 heartbeater.cc:503] Master 127.0.98.62:41889 was elected leader, sending a full tablet report...
Jun 20 00:21:04 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260620 00:21:04.599462 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.588813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42487 (local address 127.0.0.1:45512)
0620 00:21:04.589048 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:04.589082 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:04.589144 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:04.590090 (+ 946us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:04.590096 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:04.590456 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:04.590524 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.590527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.591853 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.591854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.592659 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.592662 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.593313 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.593317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.593403 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.594057 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:04.594061 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:04.595463 (+ 1402us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:04.597959 (+ 2496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.597968 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.597971 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.598227 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.598514 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.598515 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.598515 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.598584 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.599084 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:04.599092 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:04.599156 (+ 64us) client_negotiation.cc:769] Sending connection context
0620 00:21:04.599236 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:21:04.599375 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":175,"threads_started":1}
I20260620 00:21:04.599731 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.589780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.0.98.1:42487)
0620 00:21:04.589937 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.589942 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.589952 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:21:04.589986 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.589988 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.590002 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.590083 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:04.590623 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.591752 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.592739 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.593207 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.593284 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.595559 (+ 2275us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:04.595570 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:04.595570 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:04.595579 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:04.597777 (+ 2198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.598321 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.598325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.598325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.598402 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.598654 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.598655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.598655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.598966 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:04.599091 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.599344 (+ 253us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.599474 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":81,"threads_started":1}
I20260620 00:21:04.600467 16431 tablet_service.cc:1514] Processing DeleteTablet for tablet 008c15791fee4c5299c6d061bc8eeddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45512
I20260620 00:21:04.601186 16506 tablet_replica.cc:330] stopping tablet replica
I20260620 00:21:04.601321 16506 raft_consensus.cc:2229] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:04.601428 16506 raft_consensus.cc:2258] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:04.601775 16506 ts_tablet_manager.cc:1905] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:04.604269 16506 ts_tablet_manager.cc:1918] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:04.604375 16506 log.cc:1196] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal/wals/008c15791fee4c5299c6d061bc8eeddc
I20260620 00:21:04.604640 16506 ts_tablet_manager.cc:1939] T 008c15791fee4c5299c6d061bc8eeddc P f5ad9831734b4046b5dc4b177b6fbe7f: Deleting consensus metadata
I20260620 00:21:04.605211 16296 catalog_manager.cc:4841] TS f5ad9831734b4046b5dc4b177b6fbe7f (127.0.98.1:42487): tablet 008c15791fee4c5299c6d061bc8eeddc (table test-table [id=66b91ae8d7d7498ea26aea19fab21490]) successfully deleted
W20260620 00:21:04.672757 16502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:04.673043 16502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:04.673089 16502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:04.674391 16502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:04.674486 16502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:04.674538 16502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:21:04.676544 16502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35329
--tserver_master_addrs=127.0.98.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:04.676870 16502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:04.677147 16502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:04.683257 16512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:04.683269 16511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:04.683382 16502 server_base.cc:1034] running on GCE node
W20260620 00:21:04.683269 16514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:04.683871 16502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:04.684229 16502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:04.685432 16502 hybrid_clock.cc:648] HybridClock initialized: now 1781914864685425 us; error 44 us; skew 500 ppm
Jun 20 00:21:04 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:04.688544 16502 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:21:04.689081 16502 webserver.cc:460] Webserver started at http://127.0.98.2:35329/ using document root <none> and password file <none>
I20260620 00:21:04.689354 16502 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:04.689457 16502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:04.691110 16502 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:21:04.692261 16521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:04.692499 16502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:04.692713 16502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "700b63c333a74dae99c9a43b2043170e"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "685942f0f6c8aa3dda511f66f9d5733b"
server_key_iv: "4b9b9b244b26e50527cef61d71401540"
server_key_version: "encryptionkey@0"
I20260620 00:21:04.693168 16502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:04.714473 16502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:04.717226 16502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:04.717435 16502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:04.717734 16502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:04.718204 16528 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:21:04.719696 16502 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:21:04.719774 16502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260620 00:21:04.719826 16502 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:21:04.720597 16502 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:21:04.720660 16502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:21:04.720836 16528 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Bootstrap starting.
I20260620 00:21:04.728809 16502 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40979
I20260620 00:21:04.728914 16635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40979 every 8 connection(s)
I20260620 00:21:04.729269 16502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:21:04.731153 16528 log.cc:824] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:04.733474 16528 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:21:04.733795 16528 tablet_bootstrap.cc:492] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Bootstrap complete.
I20260620 00:21:04.734547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 16502
I20260620 00:21:04.735076 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45209
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41889
--webserver_port=39739
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:21:04.735110 16528 ts_tablet_manager.cc:1397] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
Jun 20 00:21:04 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:04.737002 16528 raft_consensus.cc:348] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:21:04.737315 16528 raft_consensus.cc:729] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700b63c333a74dae99c9a43b2043170e, State: Initialized, Role: FOLLOWER
I20260620 00:21:04.737525 16528 consensus_queue.cc:260] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } } peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } }
I20260620 00:21:04.737883 16528 ts_tablet_manager.cc:1428] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
I20260620 00:21:04.741178 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.730084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:50055)
0620 00:21:04.730268 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:04.730306 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:04.730860 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:04.731122 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:04.731127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:04.731515 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:04.731780 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.731786 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.732676 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.732678 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.733852 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.733855 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.734330 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.734331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.734480 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.735520 (+ 1040us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:04.735536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:04.737263 (+ 1727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:04.739490 (+ 2227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.739501 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.739504 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.739796 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.740117 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.740119 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.740120 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.740273 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.740625 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:04.740630 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:04.740769 (+ 139us) client_negotiation.cc:769] Sending connection context
0620 00:21:04.740862 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:21:04.741007 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":102,"threads_started":1}
I20260620 00:21:04.741205 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.730130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50055 (local address 127.0.98.62:41889)
0620 00:21:04.730177 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.730179 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.730402 (+ 223us) server_negotiation.cc:407] Connection header received
0620 00:21:04.730983 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.730985 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.731002 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.731058 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:04.731885 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.732554 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.733976 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.734211 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.734317 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.737413 (+ 3096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:04.737416 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:04.737416 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:04.737427 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:04.739368 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.739892 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.739894 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.739894 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.739963 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.740346 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.740347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.740348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.740534 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:04.740621 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.740935 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.741063 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:21:04.741879 16636 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41889
I20260620 00:21:04.741999 16636 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:04.742228 16636 heartbeater.cc:511] Master 127.0.98.62:41889 requested a full tablet report, sending...
I20260620 00:21:04.742969 16309 ts_manager.cc:194] Registered new tserver with Master: 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
I20260620 00:21:04.743542 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50055
I20260620 00:21:04.744494 16636 heartbeater.cc:503] Master 127.0.98.62:41889 was elected leader, sending a full tablet report...
Jun 20 00:21:04 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260620 00:21:04.752611 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.743339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40979 (local address 127.0.0.1:59908)
0620 00:21:04.743400 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:04.743411 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:04.743453 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:04.744085 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:04.744086 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:04.744322 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:04.744384 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.744387 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.745390 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.745392 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.746314 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.746317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.746929 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.746933 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.747022 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.747540 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:04.747546 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:04.749019 (+ 1473us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:04.751323 (+ 2304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.751325 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.751325 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.751534 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.751800 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.751801 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.751801 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.751886 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.752277 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:04.752279 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:04.752327 (+ 48us) client_negotiation.cc:769] Sending connection context
0620 00:21:04.752399 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:21:04.752514 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 00:21:04.752666 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.743716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59908 (local address 127.0.98.2:40979)
0620 00:21:04.743912 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.743918 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.743930 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:21:04.743968 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.743970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.743984 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.744061 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:04.744480 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.745303 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.746431 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.746748 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.746818 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.749124 (+ 2306us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:04.749136 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:04.749136 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:04.749144 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:04.751216 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.751626 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.751629 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.751630 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.751696 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.751966 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.751967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.751967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.752199 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:04.752286 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.752456 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.752562 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":133,"threads_started":1}
I20260620 00:21:04.753365 16564 tablet_service.cc:1514] Processing DeleteTablet for tablet 008c15791fee4c5299c6d061bc8eeddc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:20:44 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59908
I20260620 00:21:04.753983 16643 tablet_replica.cc:330] stopping tablet replica
I20260620 00:21:04.754120 16643 raft_consensus.cc:2229] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:04.754213 16643 raft_consensus.cc:2258] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:04.754593 16643 ts_tablet_manager.cc:1905] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:04.757102 16643 ts_tablet_manager.cc:1918] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:04.757210 16643 log.cc:1196] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/008c15791fee4c5299c6d061bc8eeddc
I20260620 00:21:04.757458 16643 ts_tablet_manager.cc:1939] T 008c15791fee4c5299c6d061bc8eeddc P 700b63c333a74dae99c9a43b2043170e: Deleting consensus metadata
I20260620 00:21:04.758085 16293 catalog_manager.cc:4841] TS 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979): tablet 008c15791fee4c5299c6d061bc8eeddc (table test-table [id=66b91ae8d7d7498ea26aea19fab21490]) successfully deleted
W20260620 00:21:04.829396 16640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:04.829694 16640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:04.829737 16640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:04.831106 16640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:04.831175 16640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:04.831223 16640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:21:04.832859 16640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39739
--tserver_master_addrs=127.0.98.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:04.833184 16640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:04.833449 16640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:04.838992 16648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:04.839023 16651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:04.838979 16649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:04.839360 16640 server_base.cc:1034] running on GCE node
I20260620 00:21:04.839578 16640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:04.839820 16640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:04.840996 16640 hybrid_clock.cc:648] HybridClock initialized: now 1781914864840977 us; error 33 us; skew 500 ppm
Jun 20 00:21:04 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:04.843943 16640 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:21:04.844519 16640 webserver.cc:460] Webserver started at http://127.0.98.3:39739/ using document root <none> and password file <none>
I20260620 00:21:04.844781 16640 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:04.844842 16640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:04.846472 16640 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:21:04.847335 16658 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:04.847640 16640 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:04.847780 16640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "cf6b2becce9a402cb864a7a3c41bf46b"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "08dcf72954432d6a0ee2709d826b57c9"
server_key_iv: "ea2ef94c403db71f6376b88d010f3c5b"
server_key_version: "encryptionkey@0"
I20260620 00:21:04.848239 16640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:04.870180 16640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:04.872877 16640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:04.873071 16640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:04.873347 16640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:04.873687 16640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:04.873747 16640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:04.873791 16640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:04.873818 16640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:04.879958 16640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45209
I20260620 00:21:04.879997 16771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45209 every 8 connection(s)
I20260620 00:21:04.880424 16640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:21:04.882141 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 16640
Jun 20 00:21:04 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:04.890609 16776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.882636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33498 (local address 127.0.98.62:41889)
0620 00:21:04.882748 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.882749 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.882853 (+ 104us) server_negotiation.cc:407] Connection header received
0620 00:21:04.882940 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.882940 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.882948 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.883000 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:04.883745 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.884404 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.885335 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.885535 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.885629 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.887107 (+ 1478us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:04.887108 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:04.887109 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:04.887115 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:04.889169 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.889645 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.889647 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.889647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.889710 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.890005 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.890007 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.890007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.890178 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:04.890352 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.890393 (+ 41us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.890481 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":60,"threads_started":1}
I20260620 00:21:04.892042 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.881162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58071 (local address 127.0.98.62:41889)
0620 00:21:04.881210 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.881213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.881542 (+ 329us) server_negotiation.cc:407] Connection header received
0620 00:21:04.882243 (+ 701us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.882244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.882258 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.882349 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:04.883754 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.884352 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.885415 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.885619 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.885706 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.888438 (+ 2732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:04.888439 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:04.888440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:04.888448 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:04.890352 (+ 1904us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.890895 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.890896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.890897 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.890966 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:04.891267 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:04.891268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:04.891268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:04.891422 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:04.891503 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.891903 (+ 400us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.891980 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:21:04.892336 16774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.881151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:58071)
0620 00:21:04.881352 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:04.881429 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:04.882088 (+ 659us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:04.882451 (+ 363us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:04.882455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:04.883354 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:04.883679 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.883685 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.884446 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.884447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.885302 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.885303 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.885719 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.885720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.885855 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.886457 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:04.886474 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:04.888329 (+ 1855us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:04.890447 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.890455 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.890458 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.890817 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.891047 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:04.891048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:04.891048 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:04.891195 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:04.891559 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:04.891567 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:04.891756 (+ 189us) client_negotiation.cc:769] Sending connection context
0620 00:21:04.891867 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:21:04.892047 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260620 00:21:04.892828 16772 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41889
I20260620 00:21:04.892947 16772 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:04.893150 16772 heartbeater.cc:511] Master 127.0.98.62:41889 requested a full tablet report, sending...
I20260620 00:21:04.893538 16309 ts_manager.cc:194] Registered new tserver with Master: cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209)
I20260620 00:21:04.893872 16309 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:58071
I20260620 00:21:04.903163 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:21:04.908353 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.903954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33502 (local address 127.0.98.62:41889)
0620 00:21:04.904057 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.904060 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.904072 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:21:04.904198 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.904199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.904259 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.904391 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:21:04.905477 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.906241 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.907367 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.907554 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.907616 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.907900 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:21:04.908010 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:21:04.908079 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.908240 (+ 161us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.908262 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260620 00:21:04.908711 16309 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:33502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:21:04.910588 16309 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:21:04.912962 16564 tablet_service.cc:1467] Processing CreateTablet for tablet dbb08c424c6c4125a99521768c0f991f (DEFAULT_TABLE table=test-table [id=e410e861e1f84c3fa188adcd25bbe934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:04.912988 16431 tablet_service.cc:1467] Processing CreateTablet for tablet dbb08c424c6c4125a99521768c0f991f (DEFAULT_TABLE table=test-table [id=e410e861e1f84c3fa188adcd25bbe934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:04.913192 16431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb08c424c6c4125a99521768c0f991f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:04.913192 16564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb08c424c6c4125a99521768c0f991f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:04.914767 16389 tablet_bootstrap.cc:492] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Bootstrap starting.
I20260620 00:21:04.915143 16389 tablet_bootstrap.cc:654] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:04.915747 16528 tablet_bootstrap.cc:492] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: Bootstrap starting.
I20260620 00:21:04.916177 16528 tablet_bootstrap.cc:654] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:04.916574 16389 tablet_bootstrap.cc:492] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: No bootstrap required, opened a new log
I20260620 00:21:04.916651 16389 ts_tablet_manager.cc:1397] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:21:04.916873 16389 raft_consensus.cc:348] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:04.916981 16389 raft_consensus.cc:374] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:04.917049 16389 raft_consensus.cc:729] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ad9831734b4046b5dc4b177b6fbe7f, State: Initialized, Role: FOLLOWER
I20260620 00:21:04.917114 16389 consensus_queue.cc:260] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:04.917261 16389 ts_tablet_manager.cc:1428] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260620 00:21:04.917671 16528 tablet_bootstrap.cc:492] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: No bootstrap required, opened a new log
I20260620 00:21:04.917752 16528 ts_tablet_manager.cc:1397] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:21:04.918103 16528 raft_consensus.cc:348] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:04.918198 16528 raft_consensus.cc:374] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:04.918270 16528 raft_consensus.cc:729] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 700b63c333a74dae99c9a43b2043170e, State: Initialized, Role: FOLLOWER
I20260620 00:21:04.918337 16528 consensus_queue.cc:260] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:04.918432 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.913038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53542 (local address 127.0.98.3:45209)
0620 00:21:04.913320 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.913326 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.914914 (+ 1588us) server_negotiation.cc:407] Connection header received
0620 00:21:04.914969 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.914972 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.914991 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.915067 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:04.915199 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.916026 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.917363 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.918029 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.918123 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.918195 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.918293 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.918331 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":107,"threads_started":1}
I20260620 00:21:04.918406 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.912791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45209 (local address 127.0.0.1:53542)
0620 00:21:04.914782 (+ 1991us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:04.914795 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:04.914847 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:04.915077 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:04.915078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:04.915080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:04.915134 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.915136 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.916181 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.916182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.917252 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.917257 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.918122 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.918123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.918186 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.918214 (+ 28us) client_negotiation.cc:769] Sending connection context
0620 00:21:04.918276 (+ 62us) client_negotiation.cc:240] Negotiation successful
0620 00:21:04.918331 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1969}
I20260620 00:21:04.918447 16528 ts_tablet_manager.cc:1428] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:04.919237 16706 tablet_service.cc:1467] Processing CreateTablet for tablet dbb08c424c6c4125a99521768c0f991f (DEFAULT_TABLE table=test-table [id=e410e861e1f84c3fa188adcd25bbe934]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:04.919586 16706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dbb08c424c6c4125a99521768c0f991f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:04.922438 16781 tablet_bootstrap.cc:492] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: Bootstrap starting.
I20260620 00:21:04.923166 16781 tablet_bootstrap.cc:654] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:04.923768 16781 log.cc:824] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:04.924717 16781 tablet_bootstrap.cc:492] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: No bootstrap required, opened a new log
I20260620 00:21:04.924795 16781 ts_tablet_manager.cc:1397] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:21:04.926966 16781 raft_consensus.cc:348] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:04.927230 16781 raft_consensus.cc:374] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:04.927258 16781 raft_consensus.cc:729] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf6b2becce9a402cb864a7a3c41bf46b, State: Initialized, Role: FOLLOWER
I20260620 00:21:04.927428 16781 consensus_queue.cc:260] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:04.927759 16781 ts_tablet_manager.cc:1428] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:21:04.927800 16772 heartbeater.cc:503] Master 127.0.98.62:41889 was elected leader, sending a full tablet report...
W20260620 00:21:04.979950 16637 tablet.cc:2369] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:04.992847 16503 raft_consensus.cc:482] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:04.993001 16503 raft_consensus.cc:504] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:04.993338 16503 leader_election.cc:288] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209), 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
I20260620 00:21:04.996431 16499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.993538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45209 (local address 127.0.98.1:43097)
0620 00:21:04.993594 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:04.993607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:04.993678 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:04.993905 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:04.993907 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:04.993909 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:04.993997 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.993999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.994661 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.994662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.995467 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.995469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.996052 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.996053 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.996182 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.996215 (+ 33us) client_negotiation.cc:769] Sending connection context
0620 00:21:04.996275 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 00:21:04.996324 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:21:04.996439 16778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.993671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:43097 (local address 127.0.98.3:45209)
0620 00:21:04.993764 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.993766 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.993780 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:21:04.993805 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.993806 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.993817 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.993899 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:04.994076 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.994563 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.995570 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.995959 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.996056 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.996067 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.996300 (+ 233us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.996348 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260620 00:21:04.996557 16784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.993588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40979 (local address 127.0.98.1:34171)
0620 00:21:04.993763 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:04.993779 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:04.993851 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:04.994071 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:04.994072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:04.994074 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:04.994146 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.994147 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.994800 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.994802 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.995558 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:04.995560 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.996295 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.996296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:04.996368 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.996371 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:04.996431 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 00:21:04.996474 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":116,"threads_started":1}
I20260620 00:21:04.996558 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:04.993654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:34171 (local address 127.0.98.2:40979)
0620 00:21:04.993705 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:04.993708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:04.993846 (+ 138us) server_negotiation.cc:407] Connection header received
0620 00:21:04.993955 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:04.993956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:04.993982 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:04.994060 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:04.994230 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.994710 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.995659 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:04.996097 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:04.996262 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:04.996353 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:04.996435 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 00:21:04.996482 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:21:04.996918 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb08c424c6c4125a99521768c0f991f" candidate_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" is_pre_election: true
I20260620 00:21:04.997114 16726 raft_consensus.cc:2454] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ad9831734b4046b5dc4b177b6fbe7f in term 0.
I20260620 00:21:04.997088 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb08c424c6c4125a99521768c0f991f" candidate_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700b63c333a74dae99c9a43b2043170e" is_pre_election: true
I20260620 00:21:04.997211 16571 raft_consensus.cc:2454] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f5ad9831734b4046b5dc4b177b6fbe7f in term 0.
I20260620 00:21:04.997388 16385 leader_election.cc:302] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cf6b2becce9a402cb864a7a3c41bf46b, f5ad9831734b4046b5dc4b177b6fbe7f; no voters:
I20260620 00:21:04.997534 16503 raft_consensus.cc:2781] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:21:04.997596 16503 raft_consensus.cc:482] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:21:04.997618 16503 raft_consensus.cc:3037] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:04.998318 16503 raft_consensus.cc:504] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:04.998482 16503 leader_election.cc:288] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 1 election: Requested vote from peers cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209), 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
I20260620 00:21:04.998623 16726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb08c424c6c4125a99521768c0f991f" candidate_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf6b2becce9a402cb864a7a3c41bf46b"
I20260620 00:21:04.998646 16571 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dbb08c424c6c4125a99521768c0f991f" candidate_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "700b63c333a74dae99c9a43b2043170e"
I20260620 00:21:04.998725 16571 raft_consensus.cc:3037] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:04.998730 16726 raft_consensus.cc:3037] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:04.999454 16571 raft_consensus.cc:2454] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ad9831734b4046b5dc4b177b6fbe7f in term 1.
I20260620 00:21:04.999478 16726 raft_consensus.cc:2454] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5ad9831734b4046b5dc4b177b6fbe7f in term 1.
I20260620 00:21:04.999644 16383 leader_election.cc:302] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 700b63c333a74dae99c9a43b2043170e, f5ad9831734b4046b5dc4b177b6fbe7f; no voters:
I20260620 00:21:04.999758 16503 raft_consensus.cc:2781] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:04.999920 16503 raft_consensus.cc:686] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 LEADER]: Becoming Leader. State: Replica: f5ad9831734b4046b5dc4b177b6fbe7f, State: Running, Role: LEADER
I20260620 00:21:05.000061 16503 consensus_queue.cc:237] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:05.001021 16309 catalog_manager.cc:5495] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f reported cstate change: term changed from 0 to 1, leader changed from <none> to f5ad9831734b4046b5dc4b177b6fbe7f (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } health_report { overall_health: HEALTHY } } }
I20260620 00:21:05.014032 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:05.011483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45514 (local address 127.0.98.1:42487)
0620 00:21:05.011549 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:05.011551 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:05.011565 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:21:05.011676 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:05.011677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:05.011696 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:05.011780 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:21:05.011897 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:05.012480 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:05.013228 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:05.013442 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:05.013531 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:05.013714 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:21:05.013771 (+ 57us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0620 00:21:05.013946 (+ 175us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45514: token expired
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 00:21:05.014106 16505 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:45514: token expired
I20260620 00:21:05.021190 16500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:05.014422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.0.98.62:41889)
0620 00:21:05.014489 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:05.014491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:05.014500 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:21:05.014528 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:05.014529 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:05.014537 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:05.014588 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:05.015063 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:05.015539 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:05.016312 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:05.016579 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:05.016635 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:05.017999 (+ 1364us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:05.018000 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:05.018001 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:05.018008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:05.019774 (+ 1766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:05.020380 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:05.020381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:05.020381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:05.020438 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:05.020672 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:05.020673 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:05.020673 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:05.020796 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:05.020889 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:05.021044 (+ 155us) server_negotiation.cc:299] Negotiation successful
0620 00:21:05.021122 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:21:05.024715 16505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:05.022229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45518 (local address 127.0.98.1:42487)
0620 00:21:05.022279 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:05.022281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:05.022299 (+ 18us) server_negotiation.cc:407] Connection header received
0620 00:21:05.022364 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:05.022365 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:05.022397 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:05.022445 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:21:05.022594 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:05.023108 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:05.023893 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:05.024102 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:05.024218 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:05.024362 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:21:05.024487 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:21:05.024556 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:05.024608 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 00:21:05.024627 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:21:05.026499 16571 raft_consensus.cc:1261] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Refusing update from remote peer f5ad9831734b4046b5dc4b177b6fbe7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:05.026499 16726 raft_consensus.cc:1261] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [term 1 FOLLOWER]: Refusing update from remote peer f5ad9831734b4046b5dc4b177b6fbe7f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:05.027017 16503 consensus_queue.cc:1035] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:05.027176 16785 consensus_queue.cc:1035] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [LEADER]: Connected to new peer: Peer: permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:05.029663 16789 mvcc.cc:205] Tried to move back new op lower bound from 7298723287145238528 to 7298723287041056768. Current Snapshot: MvccSnapshot[applied={T|T < 7298723287145238528}]
I20260620 00:21:05.031371 16790 mvcc.cc:205] Tried to move back new op lower bound from 7298723287145238528 to 7298723287041056768. Current Snapshot: MvccSnapshot[applied={T|T < 7298723287145238528}]
I20260620 00:21:05.032486 16308 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:05.032600 16308 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:05.033527 16308 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Sending DeleteTablet for 3 replicas of tablet dbb08c424c6c4125a99521768c0f991f
I20260620 00:21:05.033890 16564 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb08c424c6c4125a99521768c0f991f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59908
I20260620 00:21:05.033998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 16347
I20260620 00:21:05.033990 16706 tablet_service.cc:1514] Processing DeleteTablet for tablet dbb08c424c6c4125a99521768c0f991f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:05 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53542
I20260620 00:21:05.034202 16643 tablet_replica.cc:330] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: stopping tablet replica
I20260620 00:21:05.034265 16643 raft_consensus.cc:2229] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:05.034327 16643 raft_consensus.cc:2258] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:05.035151 16798 tablet_replica.cc:330] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: stopping tablet replica
I20260620 00:21:05.035281 16798 raft_consensus.cc:2229] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:05.035405 16798 raft_consensus.cc:2258] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:05.035535 16643 ts_tablet_manager.cc:1905] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:05.036510 16798 ts_tablet_manager.cc:1905] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:05.037529 16643 ts_tablet_manager.cc:1918] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:05.037596 16643 log.cc:1196] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/dbb08c424c6c4125a99521768c0f991f
I20260620 00:21:05.037807 16643 ts_tablet_manager.cc:1939] T dbb08c424c6c4125a99521768c0f991f P 700b63c333a74dae99c9a43b2043170e: Deleting consensus metadata
I20260620 00:21:05.037945 16798 ts_tablet_manager.cc:1918] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:05.038043 16798 log.cc:1196] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/dbb08c424c6c4125a99521768c0f991f
I20260620 00:21:05.038336 16798 ts_tablet_manager.cc:1939] T dbb08c424c6c4125a99521768c0f991f P cf6b2becce9a402cb864a7a3c41bf46b: Deleting consensus metadata
I20260620 00:21:05.038393 16293 catalog_manager.cc:4841] TS 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979): tablet dbb08c424c6c4125a99521768c0f991f (table test-table [id=e410e861e1f84c3fa188adcd25bbe934]) successfully deleted
I20260620 00:21:05.038782 16295 catalog_manager.cc:4841] TS cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209): tablet dbb08c424c6c4125a99521768c0f991f (table test-table [id=e410e861e1f84c3fa188adcd25bbe934]) successfully deleted
W20260620 00:21:05.040959 16296 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:42487 (error 108)
I20260620 00:21:05.041191 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 16502
I20260620 00:21:05.041754 16504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:05.041333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:42487 (local address 127.0.0.1:45530)
0620 00:21:05.041677 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:05.041714 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322}
W20260620 00:21:05.043166 16296 catalog_manager.cc:4584] TS f5ad9831734b4046b5dc4b177b6fbe7f (127.0.98.1:42487): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dbb08c424c6c4125a99521768c0f991f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:42487: connect: Connection refused (error 111)
I20260620 00:21:05.046808 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 16640
I20260620 00:21:05.051810 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 16276
I20260620 00:21:25.058531 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41889
--webserver_interface=127.0.98.62
--webserver_port=33261
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41889
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:25.142204 16800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:25.142442 16800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:25.142468 16800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:25.143846 16800 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260620 00:21:25.143908 16800 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260620 00:21:25.143921 16800 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260620 00:21:25.143934 16800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:25.143947 16800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:25.143961 16800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:25.144016 16800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:25.144032 16800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:25.145756 16800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41889
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41889
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=33261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16800
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:25.146066 16800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:25.146327 16800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:21:25.151979 16800 server_base.cc:1034] running on GCE node
W20260620 00:21:25.151934 16808 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:25.151927 16805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:25.151961 16806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:25.152536 16800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:25.152870 16800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:25.154021 16800 hybrid_clock.cc:648] HybridClock initialized: now 1781914885154008 us; error 31 us; skew 500 ppm
Jun 20 00:21:25 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:25.160851 16800 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:21:25.161314 16800 webserver.cc:460] Webserver started at http://127.0.98.62:33261/ using document root <none> and password file <none>
I20260620 00:21:25.161580 16800 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:25.161655 16800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:25.163137 16800 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:25.164198 16815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:25.164402 16800 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:25.164510 16800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "e7e74115cc814bebba9cc589c55f8bec"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "2234c34dc7b322dc5d01f50f320a560f"
server_key_iv: "a59f9da53af9a3ad8bb6de2195084ee5"
server_key_version: "encryptionkey@0"
I20260620 00:21:25.164899 16800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:25.178843 16800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:25.181000 16800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:25.181172 16800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:25.185496 16800 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41889
I20260620 00:21:25.185532 16867 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41889 every 8 connection(s)
I20260620 00:21:25.186009 16800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:25.187522 16868 sys_catalog.cc:263] Verifying existing consensus state
I20260620 00:21:25.188272 16868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Bootstrap starting.
I20260620 00:21:25.194257 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 16800
I20260620 00:21:25.194631 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:42487
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:41889
--webserver_port=35763
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:21:25.197104 16868 log.cc:824] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:25.199599 16868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:21:25.199941 16868 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: Bootstrap complete.
I20260620 00:21:25.202289 16868 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:21:25.202559 16868 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7e74115cc814bebba9cc589c55f8bec, State: Initialized, Role: FOLLOWER
I20260620 00:21:25.202777 16868 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:21:25.202879 16868 raft_consensus.cc:388] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:25.202934 16868 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:25.202984 16868 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 2 FOLLOWER]: Advancing to term 3
I20260620 00:21:25.204030 16868 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:21:25.204180 16868 leader_election.cc:302] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e7e74115cc814bebba9cc589c55f8bec; no voters:
I20260620 00:21:25.204417 16868 leader_election.cc:288] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [CANDIDATE]: Term 3 election: Requested vote from peers
I20260620 00:21:25.204454 16872 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 3 FOLLOWER]: Leader election won for term 3
I20260620 00:21:25.204571 16872 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [term 3 LEADER]: Becoming Leader. State: Replica: e7e74115cc814bebba9cc589c55f8bec, State: Running, Role: LEADER
I20260620 00:21:25.204661 16872 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } }
I20260620 00:21:25.204702 16868 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:25.206139 16874 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e7e74115cc814bebba9cc589c55f8bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } } }
I20260620 00:21:25.206234 16874 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:25.206435 16873 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7e74115cc814bebba9cc589c55f8bec. Latest consensus state: current_term: 3 leader_uuid: "e7e74115cc814bebba9cc589c55f8bec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e7e74115cc814bebba9cc589c55f8bec" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41889 } } }
I20260620 00:21:25.206483 16873 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:25.206856 16886 catalog_manager.cc:1259] Loaded cluster ID: 69dcc52e280744e982fcceebe960f272
I20260620 00:21:25.206910 16886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: loading cluster ID for follower catalog manager: success
I20260620 00:21:25.207687 16886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e7e74115cc814bebba9cc589c55f8bec: acquiring CA information for follower catalog manager: success
F20260620 00:21:25.208051 16886 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f8c0b8259ad google::LogMessage::Fail() at ??:0
@ 0x7f8c0b82888a google::LogMessage::SendToLog() at ??:0
@ 0x7f8c0b8254de google::LogMessage::Flush() at ??:0
@ 0x7f8c0b826a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f8c0df92104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f8c137826b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7f8c13782989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7f8c13782e4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7f8c137858c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7f8c0d6e8954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f8c0e54c6db start_thread at ??:0
@ 0x7f8c0a32361f clone at ??:0
W20260620 00:21:25.293691 16870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:25.293941 16870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:25.293982 16870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:25.295236 16870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:25.295305 16870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:25.295365 16870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:21:25.296988 16870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:42487
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35763
--tserver_master_addrs=127.0.98.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:25.297279 16870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:25.297523 16870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:25.302932 16909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:25.302906 16906 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:25.302915 16907 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:25.303222 16870 server_base.cc:1034] running on GCE node
I20260620 00:21:25.303435 16870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:25.303699 16870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:25.304982 16870 hybrid_clock.cc:648] HybridClock initialized: now 1781914885304952 us; error 47 us; skew 500 ppm
Jun 20 00:21:25 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:25.308005 16870 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:21:25.308580 16870 webserver.cc:460] Webserver started at http://127.0.98.1:35763/ using document root <none> and password file <none>
I20260620 00:21:25.308837 16870 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:25.308910 16870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:25.310510 16870 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260620 00:21:25.311407 16916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:25.311645 16870 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:25.311792 16870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "f5ad9831734b4046b5dc4b177b6fbe7f"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "58cd42becbbe9cc1832dc3829e342e96"
server_key_iv: "8bd624fea58e3294b2780c34bc0537bd"
server_key_version: "encryptionkey@0"
I20260620 00:21:25.312234 16870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:25.326531 16870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:25.329545 16870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:25.329727 16870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:25.330014 16870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:25.330523 16923 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260620 00:21:25.333005 16870 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260620 00:21:25.333179 16870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.003s user 0.000s sys 0.000s
I20260620 00:21:25.333230 16870 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260620 00:21:25.333966 16870 ts_tablet_manager.cc:610] Registered 1 tablets
I20260620 00:21:25.334028 16870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260620 00:21:25.334141 16923 tablet_bootstrap.cc:492] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Bootstrap starting.
I20260620 00:21:25.342238 16870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:42487
I20260620 00:21:25.342291 17030 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:42487 every 8 connection(s)
I20260620 00:21:25.342722 16870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:21:25.349628 16923 log.cc:824] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:25.350581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 16870
I20260620 00:21:25.350935 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:40979
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:41889
--webserver_port=35329
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:21:25.353036 16923 tablet_bootstrap.cc:492] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260620 00:21:25.353466 16923 tablet_bootstrap.cc:492] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Bootstrap complete.
I20260620 00:21:25.355024 16923 ts_tablet_manager.cc:1397] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.011s
I20260620 00:21:25.357182 16923 raft_consensus.cc:348] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:25.357590 16923 raft_consensus.cc:729] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f5ad9831734b4046b5dc4b177b6fbe7f, State: Initialized, Role: FOLLOWER
I20260620 00:21:25.357831 16923 consensus_queue.cc:260] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:25.358191 16923 ts_tablet_manager.cc:1428] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f: Time spent starting tablet: real 0.003s user 0.006s sys 0.000s
W20260620 00:21:25.437469 17035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:25.437741 17035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:25.437788 17035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:25.439078 17035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:25.439263 17035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:25.439353 17035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:21:25.441244 17035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:40979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=35329
--tserver_master_addrs=127.0.98.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:25.441566 17035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:25.441828 17035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:25.447352 17042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:25.447321 17041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:25.447485 17035 server_base.cc:1034] running on GCE node
W20260620 00:21:25.447335 17044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:25.447831 17035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:25.448160 17035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:25.449358 17035 hybrid_clock.cc:648] HybridClock initialized: now 1781914885449339 us; error 34 us; skew 500 ppm
Jun 20 00:21:25 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:25.453120 17035 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:21:25.453671 17035 webserver.cc:460] Webserver started at http://127.0.98.2:35329/ using document root <none> and password file <none>
I20260620 00:21:25.453926 17035 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:25.453999 17035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:25.455436 17035 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:21:25.456192 17051 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:25.456379 17035 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
W20260620 00:21:25.456404 17033 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:21:25.343578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:48965)
0620 00:21:25.343993 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:25.344042 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:25.344631 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:25.456192 (+111561us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":246,"threads_started":1}
W20260620 00:21:25.456652 17031 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41889 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41889: Client connection negotiation failed: client connection to 127.0.98.62:41889: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260620 00:21:25.456465 17035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "700b63c333a74dae99c9a43b2043170e"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "685942f0f6c8aa3dda511f66f9d5733b"
server_key_iv: "4b9b9b244b26e50527cef61d71401540"
server_key_version: "encryptionkey@0"
I20260620 00:21:25.456987 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:25.456870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:58099)
0620 00:21:25.456922 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:25.456960 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260620 00:21:25.457001 17035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:25.457324 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:25.457232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:47133)
0620 00:21:25.457278 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:25.457302 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:21:25.457469 17031 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:21:25.464270 17035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:25.466935 17035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:25.467064 17035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:25.467345 17035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:25.467707 17035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:25.467773 17035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:25.467819 17035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:25.467851 17035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:25.474217 17035 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:40979
I20260620 00:21:25.474263 17164 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:40979 every 8 connection(s)
I20260620 00:21:25.474633 17035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:21:25.475890 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:25.475375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:35431)
0620 00:21:25.475587 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:25.475700 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":88,"threads_started":1}
W20260620 00:21:25.476414 17165 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41889 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41889: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
I20260620 00:21:25.476519 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 17035
I20260620 00:21:25.476801 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:25.476670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:43333)
0620 00:21:25.476732 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:25.476765 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260620 00:21:25.476936 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:45209
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:41889
--webserver_port=39739
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:21:25.477142 17167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:25.477035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:54263)
0620 00:21:25.477089 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:25.477118 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260620 00:21:25.477303 17165 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260620 00:21:25.566219 17168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:25.566497 17168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:25.566541 17168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:25.567781 17168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:25.567866 17168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:25.567914 17168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:21:25.569775 17168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:45209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39739
--tserver_master_addrs=127.0.98.62:41889
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:25.570128 17168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:25.570408 17168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:25.575874 17176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:25.575884 17174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:25.575884 17173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:25.576208 17168 server_base.cc:1034] running on GCE node
I20260620 00:21:25.576480 17168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:25.576768 17168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:25.577936 17168 hybrid_clock.cc:648] HybridClock initialized: now 1781914885577916 us; error 38 us; skew 500 ppm
Jun 20 00:21:25 dist-test-slave-r81b krb5kdc[15728](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:25.580890 17168 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:21:25.581431 17168 webserver.cc:460] Webserver started at http://127.0.98.3:39739/ using document root <none> and password file <none>
I20260620 00:21:25.581689 17168 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:25.581761 17168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:25.583201 17168 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:25.583936 17183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:25.584118 17168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:25.584290 17168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "cf6b2becce9a402cb864a7a3c41bf46b"
format_stamp: "Formatted at 2026-06-20 00:20:43 on dist-test-slave-r81b"
server_key: "08dcf72954432d6a0ee2709d826b57c9"
server_key_iv: "ea2ef94c403db71f6376b88d010f3c5b"
server_key_version: "encryptionkey@0"
I20260620 00:21:25.584673 17168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:25.616300 17168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:25.618906 17168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:25.619120 17168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:25.619421 17168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:25.619843 17168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:25.619910 17168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:25.619958 17168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:25.619988 17168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:25.626571 17168 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45209
I20260620 00:21:25.626614 17296 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45209 every 8 connection(s)
I20260620 00:21:25.626986 17168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:21:25.628062 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:25.627660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:36845)
0620 00:21:25.627841 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:25.627918 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":79,"threads_started":1}
W20260620 00:21:25.628556 17297 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:41889 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:41889: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
I20260620 00:21:25.629050 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:25.628903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:38689)
0620 00:21:25.628991 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:25.629026 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 00:21:25.629410 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:25.629307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:54327)
0620 00:21:25.629334 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:25.629356 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260620 00:21:25.629546 17297 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260620 00:21:25.633103 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 17168
I20260620 00:21:26.458132 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:26.457898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:34429)
0620 00:21:26.458041 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:26.458077 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":60,"threads_started":1}
I20260620 00:21:26.478145 17302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:26.477749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:36705)
0620 00:21:26.478015 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:26.478065 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":151,"threads_started":1}
I20260620 00:21:26.630199 17303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:26.629903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:41331)
0620 00:21:26.630053 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:26.630104 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":57,"threads_started":1}
I20260620 00:21:26.957598 17304 raft_consensus.cc:482] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:26.957711 17304 raft_consensus.cc:504] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:26.958108 17304 leader_election.cc:288] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209), 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
Jun 20 00:21:26 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Jun 20 00:21:26 dist-test-slave-r81b krb5kdc[15728](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260620 00:21:26.970290 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:26.958242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45209 (local address 127.0.98.1:59571)
0620 00:21:26.958306 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:26.958317 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:26.958375 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:26.959637 (+ 1262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:26.959638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:26.959981 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:26.960288 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:26.960297 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:26.961506 (+ 1209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:26.961508 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:26.962427 (+ 919us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:26.962429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:26.963381 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:26.963383 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:26.963497 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:26.964118 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:26.964130 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:26.966359 (+ 2229us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:26.968631 (+ 2272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:26.968641 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:26.968643 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:26.968895 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:26.969175 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:26.969176 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:26.969177 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:26.969298 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:26.969724 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:26.969731 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:26.969881 (+ 150us) client_negotiation.cc:769] Sending connection context
0620 00:21:26.969992 (+ 111us) client_negotiation.cc:240] Negotiation successful
0620 00:21:26.970169 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":1}
I20260620 00:21:26.970314 17306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:26.958386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59571 (local address 127.0.98.3:45209)
0620 00:21:26.958596 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:26.958611 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:26.958648 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:21:26.959447 (+ 799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:26.959450 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:26.959508 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:26.959664 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:26.960444 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:26.961385 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:26.962520 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:26.963268 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:26.963779 (+ 511us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:26.966485 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:26.966489 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:26.966489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:26.966500 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:26.968517 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:26.968995 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:26.969005 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:26.969006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:26.969097 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:26.969369 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:26.969370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:26.969371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:26.969640 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:26.969732 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:26.970046 (+ 314us) server_negotiation.cc:299] Negotiation successful
0620 00:21:26.970169 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":44,"threads_started":1}
I20260620 00:21:26.970698 17305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:26.958311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:40979 (local address 127.0.98.1:51607)
0620 00:21:26.958461 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:26.958471 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:26.958525 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:26.959485 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:26.959488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:26.959981 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:26.960288 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:26.960312 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:26.962162 (+ 1850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:26.962163 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:26.962741 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:26.962743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:26.963358 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:26.963359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:26.963491 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:26.964196 (+ 705us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:26.964200 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:26.965900 (+ 1700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:26.969143 (+ 3243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:26.969144 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:26.969144 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:26.969385 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:26.969703 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:26.969704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:26.969705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:26.969771 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:26.970313 (+ 542us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:26.970314 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:26.970368 (+ 54us) client_negotiation.cc:769] Sending connection context
0620 00:21:26.970419 (+ 51us) client_negotiation.cc:240] Negotiation successful
0620 00:21:26.970547 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":27,"thread_start_us":99,"threads_started":1}
I20260620 00:21:26.971208 17307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:26.958557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51607 (local address 127.0.98.2:40979)
0620 00:21:26.958706 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:26.958717 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:26.958750 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:21:26.959169 (+ 419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:26.959171 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:26.959243 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:26.959428 (+ 185us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:26.960404 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:26.962057 (+ 1653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:26.962824 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:26.963239 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:26.963389 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:26.966033 (+ 2644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:26.966040 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:26.966040 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:26.966459 (+ 419us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:26.969029 (+ 2570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:26.969498 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:26.969501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:26.969502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:26.969626 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:26.969844 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:26.969844 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:26.969845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:26.970208 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:26.970796 (+ 588us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:26.970898 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:21:26.971057 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":91,"threads_started":1}
W20260620 00:21:26.971822 16919 leader_election.cc:341] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209): Not found: Tablet not found: dbb08c424c6c4125a99521768c0f991f
W20260620 00:21:26.971915 16917 leader_election.cc:341] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979): Not found: Tablet not found: dbb08c424c6c4125a99521768c0f991f
I20260620 00:21:26.971997 16917 leader_election.cc:302] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5ad9831734b4046b5dc4b177b6fbe7f; no voters: 700b63c333a74dae99c9a43b2043170e, cf6b2becce9a402cb864a7a3c41bf46b
I20260620 00:21:26.972116 17304 raft_consensus.cc:2726] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:21:27.458818 17301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:27.458652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:37893)
0620 00:21:27.458718 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:27.458760 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:21:27.478953 17308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:27.478662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:37959)
0620 00:21:27.478843 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:27.478911 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":77,"threads_started":1}
I20260620 00:21:27.630987 17309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:27.630737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:38199)
0620 00:21:27.630880 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:27.630932 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260620 00:21:28.459915 17310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:28.459497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:52087)
0620 00:21:28.459822 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:28.459864 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":297,"thread_start_us":81,"threads_started":1}
I20260620 00:21:28.479758 17311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:28.479486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:34187)
0620 00:21:28.479664 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:28.479712 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":75,"threads_started":1}
I20260620 00:21:28.631819 17312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:28.631575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:47099)
0620 00:21:28.631727 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:28.631771 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260620 00:21:28.851553 17313 raft_consensus.cc:482] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:28.851701 17313 raft_consensus.cc:504] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:28.851958 17313 leader_election.cc:288] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209), 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
W20260620 00:21:28.852545 16917 leader_election.cc:341] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979): Not found: Tablet not found: dbb08c424c6c4125a99521768c0f991f
W20260620 00:21:28.852618 16919 leader_election.cc:341] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209): Not found: Tablet not found: dbb08c424c6c4125a99521768c0f991f
I20260620 00:21:28.852650 16919 leader_election.cc:302] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5ad9831734b4046b5dc4b177b6fbe7f; no voters: 700b63c333a74dae99c9a43b2043170e, cf6b2becce9a402cb864a7a3c41bf46b
I20260620 00:21:28.852723 17313 raft_consensus.cc:2726] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:21:29.460939 17314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:29.460626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:55373)
0620 00:21:29.460844 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:29.460890 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":154,"threads_started":1}
I20260620 00:21:29.480748 17315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:29.480471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:54071)
0620 00:21:29.480620 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:29.480697 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":94,"threads_started":1}
I20260620 00:21:29.632704 17316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:29.632447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:54267)
0620 00:21:29.632607 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:29.632646 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":74,"threads_started":1}
I20260620 00:21:30.461830 17317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:30.461552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:51457)
0620 00:21:30.461733 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:30.461781 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
I20260620 00:21:30.481555 17318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:30.481300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:51561)
0620 00:21:30.481446 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:30.481494 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
I20260620 00:21:30.633491 17319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:30.633214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:52885)
0620 00:21:30.633395 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:30.633443 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":77,"threads_started":1}
I20260620 00:21:30.785653 17320 raft_consensus.cc:482] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:30.785810 17320 raft_consensus.cc:504] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:30.786012 17320 leader_election.cc:288] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209), 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
W20260620 00:21:30.786440 16919 leader_election.cc:341] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209): Not found: Tablet not found: dbb08c424c6c4125a99521768c0f991f
W20260620 00:21:30.786520 16917 leader_election.cc:341] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979): Not found: Tablet not found: dbb08c424c6c4125a99521768c0f991f
I20260620 00:21:30.786586 16917 leader_election.cc:302] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5ad9831734b4046b5dc4b177b6fbe7f; no voters: 700b63c333a74dae99c9a43b2043170e, cf6b2becce9a402cb864a7a3c41bf46b
I20260620 00:21:30.786679 17320 raft_consensus.cc:2726] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:21:31.462709 17321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:31.462447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:59675)
0620 00:21:31.462595 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:31.462658 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":65,"threads_started":1}
I20260620 00:21:31.482391 17322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:31.482139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:45137)
0620 00:21:31.482306 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:31.482343 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":100,"threads_started":1}
I20260620 00:21:31.634377 17323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:31.634092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:41119)
0620 00:21:31.634273 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:31.634327 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":70,"threads_started":1}
I20260620 00:21:32.463688 17324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:32.463405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:44083)
0620 00:21:32.463577 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:32.463627 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":68,"threads_started":1}
I20260620 00:21:32.483266 17325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:32.483000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:48055)
0620 00:21:32.483172 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:32.483217 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":121,"threads_started":1}
I20260620 00:21:32.635324 17326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:32.635036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:59517)
0620 00:21:32.635229 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:32.635276 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":93,"threads_started":1}
I20260620 00:21:33.464792 17327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:33.464488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:44751)
0620 00:21:33.464688 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:33.464741 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":86,"threads_started":1}
I20260620 00:21:33.484108 17328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:33.483863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:38071)
0620 00:21:33.484012 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:33.484061 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260620 00:21:33.636329 17329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:33.636024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:44039)
0620 00:21:33.636218 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:33.636274 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":64,"threads_started":1}
I20260620 00:21:34.465762 17330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:34.465503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:53431)
0620 00:21:34.465662 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:34.465712 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":69,"threads_started":1}
I20260620 00:21:34.484923 17331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:34.484682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:43735)
0620 00:21:34.484837 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:34.484878 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":73,"threads_started":1}
I20260620 00:21:34.637223 17332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:34.636944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:60769)
0620 00:21:34.637124 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:34.637171 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":86,"threads_started":1}
I20260620 00:21:35.317720 17333 raft_consensus.cc:482] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:35.317855 17333 raft_consensus.cc:504] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cf6b2becce9a402cb864a7a3c41bf46b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45209 } } peers { permanent_uuid: "700b63c333a74dae99c9a43b2043170e" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 40979 } } peers { permanent_uuid: "f5ad9831734b4046b5dc4b177b6fbe7f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 42487 } }
I20260620 00:21:35.318105 17333 leader_election.cc:288] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209), 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979)
W20260620 00:21:35.318616 16919 leader_election.cc:341] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer cf6b2becce9a402cb864a7a3c41bf46b (127.0.98.3:45209): Not found: Tablet not found: dbb08c424c6c4125a99521768c0f991f
W20260620 00:21:35.318816 16917 leader_election.cc:341] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 700b63c333a74dae99c9a43b2043170e (127.0.98.2:40979): Not found: Tablet not found: dbb08c424c6c4125a99521768c0f991f
I20260620 00:21:35.318892 16917 leader_election.cc:302] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f5ad9831734b4046b5dc4b177b6fbe7f; no voters: 700b63c333a74dae99c9a43b2043170e, cf6b2becce9a402cb864a7a3c41bf46b
I20260620 00:21:35.318986 17333 raft_consensus.cc:2726] T dbb08c424c6c4125a99521768c0f991f P f5ad9831734b4046b5dc4b177b6fbe7f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260620 00:21:35.466686 17334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:35.466432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:44519)
0620 00:21:35.466585 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:35.466634 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":67,"threads_started":1}
I20260620 00:21:35.485788 17335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:35.485538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:38701)
0620 00:21:35.485700 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:35.485738 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":100,"threads_started":1}
I20260620 00:21:35.638409 17336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:35.638047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:50321)
0620 00:21:35.638303 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:35.638360 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":76,"threads_started":1}
I20260620 00:21:36.467531 17337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:36.467267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:56677)
0620 00:21:36.467433 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:36.467480 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260620 00:21:36.486495 17338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:36.486277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:34697)
0620 00:21:36.486403 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:36.486447 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":63,"threads_started":1}
I20260620 00:21:36.639382 17339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:36.639089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:45291)
0620 00:21:36.639285 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:36.639331 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":81,"threads_started":1}
I20260620 00:21:37.468743 17340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:37.468432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:52179)
0620 00:21:37.468582 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:37.468632 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":69,"threads_started":1}
I20260620 00:21:37.487378 17341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:37.487116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:58403)
0620 00:21:37.487282 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:37.487328 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":62,"threads_started":1}
I20260620 00:21:37.640234 17342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:37.639913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:34403)
0620 00:21:37.640097 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:37.640170 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":106,"threads_started":1}
I20260620 00:21:38.469574 17343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:38.469336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:56393)
0620 00:21:38.469479 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:38.469525 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":68,"threads_started":1}
I20260620 00:21:38.488380 17344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:38.488081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:37005)
0620 00:21:38.488275 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:38.488318 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":127,"threads_started":1}
I20260620 00:21:38.640971 17345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:38.640731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:46763)
0620 00:21:38.640878 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:38.640923 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":63,"threads_started":1}
I20260620 00:21:39.470467 17346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:39.470216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:53781)
0620 00:21:39.470376 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:39.470418 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":82,"threads_started":1}
I20260620 00:21:39.489319 17347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:39.488979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:42615)
0620 00:21:39.489186 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:39.489235 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":114,"threads_started":1}
I20260620 00:21:39.641808 17348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:39.641528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:37991)
0620 00:21:39.641715 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:39.641758 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20260620 00:21:40.471643 17349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:40.471283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.1:60795)
0620 00:21:40.471477 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:40.471525 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":82,"threads_started":1}
I20260620 00:21:40.490211 17350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:40.489960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.2:39829)
0620 00:21:40.490103 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:40.490164 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260620 00:21:40.642895 17351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:40.642554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41889 (local address 127.0.98.3:60663)
0620 00:21:40.642741 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:40.642828 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:41889: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":113,"threads_started":1}
I20260620 00:21:40.643061 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 16870
I20260620 00:21:40.648398 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 17035
I20260620 00:21:40.653234 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 17168
2026-06-20T00:21:40Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58490 ms)
[----------] 30 tests from SecurityITest (233742 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:40 dist-test-slave-r81b krb5kdc[17354](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:40 dist-test-slave-r81b krb5kdc[17354](info): set up 2 sockets
Jun 20 00:21:40 dist-test-slave-r81b krb5kdc[17354](info): commencing operation
krb5kdc: starting...
W20260620 00:21:41.716070 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:41 dist-test-slave-r81b krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:21:41.847780 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37813
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37813
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:41.932082 17367 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:41.932368 17367 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:41.932395 17367 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:41.933729 17367 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:41.933804 17367 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:41.933831 17367 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:41.933845 17367 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:41.933858 17367 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:41.935673 17367 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37813
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17367
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:41.935911 17367 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:41.936120 17367 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:21:41.941242 17367 server_base.cc:1034] running on GCE node
W20260620 00:21:41.941227 17373 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:41.941212 17372 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:41.941223 17375 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:41.941694 17367 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:41.942015 17367 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:41.943177 17367 hybrid_clock.cc:648] HybridClock initialized: now 1781914901943147 us; error 48 us; skew 500 ppm
Jun 20 00:21:41 dist-test-slave-r81b krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:41.946043 17367 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:21:41.946463 17367 webserver.cc:460] Webserver started at http://127.0.98.62:42209/ using document root <none> and password file <none>
I20260620 00:21:41.946657 17367 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:41.946699 17367 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:41.946791 17367 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:41.947862 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "a50244bf8ad747f4983c523a7fe77152"
format_stamp: "Formatted at 2026-06-20 00:21:41 on dist-test-slave-r81b"
server_key: "f598172de54762e636be196d98f6fcf7"
server_key_iv: "e9f7c342f56f47bf940e8786b3da28ca"
server_key_version: "encryptionkey@0"
I20260620 00:21:41.948235 17367 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "a50244bf8ad747f4983c523a7fe77152"
format_stamp: "Formatted at 2026-06-20 00:21:41 on dist-test-slave-r81b"
server_key: "f598172de54762e636be196d98f6fcf7"
server_key_iv: "e9f7c342f56f47bf940e8786b3da28ca"
server_key_version: "encryptionkey@0"
I20260620 00:21:41.949648 17367 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:41.950486 17382 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:41.950820 17367 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:41.950941 17367 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "a50244bf8ad747f4983c523a7fe77152"
format_stamp: "Formatted at 2026-06-20 00:21:41 on dist-test-slave-r81b"
server_key: "f598172de54762e636be196d98f6fcf7"
server_key_iv: "e9f7c342f56f47bf940e8786b3da28ca"
server_key_version: "encryptionkey@0"
I20260620 00:21:41.951027 17367 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:41.959758 17367 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:41.962169 17367 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:41.962344 17367 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:41.966657 17367 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37813
I20260620 00:21:41.966696 17434 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37813 every 8 connection(s)
I20260620 00:21:41.967087 17367 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:41.968109 17435 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:41.970568 17435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152: Bootstrap starting.
I20260620 00:21:41.971136 17435 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:41.971376 17435 log.cc:824] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:41.972277 17435 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152: No bootstrap required, opened a new log
I20260620 00:21:41.973307 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 17367
I20260620 00:21:41.973430 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:41.973659 392 external_mini_cluster.cc:1428] Setting key dfb23d07cf6d48cc1c943347b2dcd6dd
I20260620 00:21:41.974128 17435 raft_consensus.cc:348] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50244bf8ad747f4983c523a7fe77152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260620 00:21:41.974445 17435 raft_consensus.cc:374] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:41.974527 17435 raft_consensus.cc:729] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a50244bf8ad747f4983c523a7fe77152, State: Initialized, Role: FOLLOWER
I20260620 00:21:41.974704 17435 consensus_queue.cc:260] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50244bf8ad747f4983c523a7fe77152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260620 00:21:41.974819 17435 raft_consensus.cc:388] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:41.974857 17435 raft_consensus.cc:482] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:41.974943 17435 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:41.976233 17435 raft_consensus.cc:504] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50244bf8ad747f4983c523a7fe77152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260620 00:21:41.976342 17435 leader_election.cc:302] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a50244bf8ad747f4983c523a7fe77152; no voters:
I20260620 00:21:41.976526 17435 leader_election.cc:288] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:41.976552 17440 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:41.976724 17440 raft_consensus.cc:686] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [term 1 LEADER]: Becoming Leader. State: Replica: a50244bf8ad747f4983c523a7fe77152, State: Running, Role: LEADER
I20260620 00:21:41.976809 17435 sys_catalog.cc:564] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:41.976907 17440 consensus_queue.cc:237] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50244bf8ad747f4983c523a7fe77152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } }
I20260620 00:21:41.977275 17440 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a50244bf8ad747f4983c523a7fe77152. Latest consensus state: current_term: 1 leader_uuid: "a50244bf8ad747f4983c523a7fe77152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50244bf8ad747f4983c523a7fe77152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } } }
I20260620 00:21:41.977538 17440 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:41.977458 17442 sys_catalog.cc:455] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a50244bf8ad747f4983c523a7fe77152" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a50244bf8ad747f4983c523a7fe77152" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37813 } } }
I20260620 00:21:41.977698 17442 sys_catalog.cc:458] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:41.977854 17449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:41.978456 17449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:21:41 dist-test-slave-r81b krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914901, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:41.982537 17449 catalog_manager.cc:1347] Generated new cluster ID: c9d0edea3e0b4496aadcd10b3b13d211
I20260620 00:21:41.982633 17449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:41.986840 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:41.974257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57806 (local address 127.0.98.62:37813)
0620 00:21:41.974446 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:41.974455 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:41.974483 (+ 28us) server_negotiation.cc:407] Connection header received
0620 00:21:41.974622 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:41.974631 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:41.974690 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:41.974826 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:41.975693 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:41.977147 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:41.978388 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:41.978982 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:41.979178 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:41.982516 (+ 3338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:41.982520 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:41.982521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:41.982534 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:41.984842 (+ 2308us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:41.985325 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:41.985329 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:41.985330 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:41.985527 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:41.985760 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:41.985761 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:41.985762 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:41.986149 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:41.986326 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:41.986416 (+ 90us) server_negotiation.cc:299] Negotiation successful
0620 00:21:41.986588 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
I20260620 00:21:42.001021 17449 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:42.001673 17449 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:42.006129 17449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152: Generated new TSK 0
I20260620 00:21:42.006394 17449 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:21:42.045213 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:42.133411 17461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:42.133658 17461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:42.133688 17461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:42.135061 17461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:42.135118 17461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:42.135155 17461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:21:42.136868 17461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:42.137120 17461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:42.137332 17461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:42.143051 17469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:42.143116 17467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:42.143051 17466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:42.143103 17461 server_base.cc:1034] running on GCE node
I20260620 00:21:42.143621 17461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:42.143926 17461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:42.145151 17461 hybrid_clock.cc:648] HybridClock initialized: now 1781914902145137 us; error 32 us; skew 500 ppm
Jun 20 00:21:42 dist-test-slave-r81b krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:42.148032 17461 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:21:42.148630 17461 webserver.cc:460] Webserver started at http://127.0.98.1:43419/ using document root <none> and password file <none>
I20260620 00:21:42.148936 17461 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:42.149013 17461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:42.149170 17461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:42.150287 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "3cdf1d3f40d240d4b2c25901bff8ca05"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "2a00792b2bdd37964e70f12fe9579f0d"
server_key_iv: "f18a10512e0f351cea26e68c9445c892"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.150658 17461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "3cdf1d3f40d240d4b2c25901bff8ca05"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "2a00792b2bdd37964e70f12fe9579f0d"
server_key_iv: "f18a10512e0f351cea26e68c9445c892"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.152060 17461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:21:42.152957 17476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.153188 17461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:42.153321 17461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "3cdf1d3f40d240d4b2c25901bff8ca05"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "2a00792b2bdd37964e70f12fe9579f0d"
server_key_iv: "f18a10512e0f351cea26e68c9445c892"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.153409 17461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:42.168756 17461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:42.171023 17461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:42.171209 17461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:42.171464 17461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:42.171802 17461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:42.171860 17461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.171898 17461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:42.171924 17461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.178610 17461 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43961
I20260620 00:21:42.178800 17589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43961 every 8 connection(s)
I20260620 00:21:42.179100 17461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:21:42.180958 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 17461
I20260620 00:21:42.181088 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:21:42.181288 392 external_mini_cluster.cc:1428] Setting key 002a530101f71dbc645adb05c37db527
Jun 20 00:21:42 dist-test-slave-r81b krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260620 00:21:42.191246 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.179950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50795 (local address 127.0.98.62:37813)
0620 00:21:42.179998 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.180000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.180271 (+ 271us) server_negotiation.cc:407] Connection header received
0620 00:21:42.181419 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.181421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.181433 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.181489 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:42.182320 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.182835 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.184346 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.184606 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.184693 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.187442 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:42.187444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:42.187445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:42.187458 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:42.189412 (+ 1954us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:42.189955 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:42.189957 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:42.189958 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:42.190028 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:42.190405 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:42.190407 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:42.190407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:42.190567 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:42.190645 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.191012 (+ 367us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.191139 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:21:42.191440 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.179921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37813 (local address 127.0.98.1:50795)
0620 00:21:42.180105 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:42.180166 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:42.181295 (+ 1129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:42.181581 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:42.181585 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:42.181986 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:42.182217 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.182222 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.182939 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.182940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.184228 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.184232 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.184754 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.184755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.184904 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.185570 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:42.185580 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:42.187304 (+ 1724us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:42.189537 (+ 2233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:42.189547 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:42.189550 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:42.189862 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:42.190168 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:42.190169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:42.190169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:42.190317 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:42.190654 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:42.190662 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:42.190811 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 00:21:42.190968 (+ 157us) client_negotiation.cc:240] Negotiation successful
0620 00:21:42.191172 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":85,"threads_started":1}
I20260620 00:21:42.192094 17590 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37813
I20260620 00:21:42.192317 17590 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:42.192536 17590 heartbeater.cc:511] Master 127.0.98.62:37813 requested a full tablet report, sending...
I20260620 00:21:42.193281 17399 ts_manager.cc:194] Registered new tserver with Master: 3cdf1d3f40d240d4b2c25901bff8ca05 (127.0.98.1:43961)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260620 00:21:42.194208 17399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:50795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:21:42.212703 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:42.300716 17595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:42.300959 17595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:42.300985 17595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:42.302285 17595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:42.302345 17595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:42.302376 17595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:21:42.304036 17595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:42.304313 17595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:42.304596 17595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:42.309993 17601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:42.310035 17603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:42.310009 17600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:42.310446 17595 server_base.cc:1034] running on GCE node
I20260620 00:21:42.310725 17595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:42.310961 17595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:42.312187 17595 hybrid_clock.cc:648] HybridClock initialized: now 1781914902312116 us; error 32 us; skew 500 ppm
Jun 20 00:21:42 dist-test-slave-r81b krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:42.315443 17595 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:21:42.315999 17595 webserver.cc:460] Webserver started at http://127.0.98.2:38087/ using document root <none> and password file <none>
I20260620 00:21:42.316344 17595 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:42.316418 17595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:42.316572 17595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:42.317682 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "797768b2f72b4c9cb5fba707b0564bff"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "091ce1afc77b530e887514c7a8cd69a2"
server_key_iv: "423085174ab6190264d61208ea4a2023"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.318081 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "797768b2f72b4c9cb5fba707b0564bff"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "091ce1afc77b530e887514c7a8cd69a2"
server_key_iv: "423085174ab6190264d61208ea4a2023"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.319494 17595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:21:42.320413 17610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.320726 17595 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:42.320860 17595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "797768b2f72b4c9cb5fba707b0564bff"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "091ce1afc77b530e887514c7a8cd69a2"
server_key_iv: "423085174ab6190264d61208ea4a2023"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.320964 17595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:42.329294 17595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:42.331584 17595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:42.331815 17595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:42.332078 17595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:42.332475 17595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:42.332533 17595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.332577 17595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:42.332609 17595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.338959 17595 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35467
I20260620 00:21:42.339018 17723 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35467 every 8 connection(s)
I20260620 00:21:42.339359 17595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:21:42.343303 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 17595
I20260620 00:21:42.343415 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:21:42.343657 392 external_mini_cluster.cc:1428] Setting key 2336cb85ed517924a25f3eed82e74388
Jun 20 00:21:42 dist-test-slave-r81b krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:42.350829 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.340197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56493 (local address 127.0.98.62:37813)
0620 00:21:42.340246 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.340249 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.340463 (+ 214us) server_negotiation.cc:407] Connection header received
0620 00:21:42.341529 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.341530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.341544 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.341595 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:42.342290 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.342810 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.344129 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.344377 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.344467 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.347346 (+ 2879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:42.347348 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:42.347348 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:42.347357 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:42.349174 (+ 1817us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:42.349641 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:42.349643 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:42.349644 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:42.349714 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:42.350056 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:42.350058 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:42.350058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:42.350224 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:42.350314 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.350613 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.350713 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:21:42.350943 17726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.340097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37813 (local address 127.0.98.2:56493)
0620 00:21:42.340318 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:42.340371 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:42.341395 (+ 1024us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:42.341671 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:42.341676 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:42.341977 (+ 301us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:42.342178 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.342183 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.342917 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.342918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.344043 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.344046 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.344493 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.344494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.344679 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.345262 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:42.345272 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:42.347223 (+ 1951us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:42.349272 (+ 2049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:42.349292 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:42.349295 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:42.349551 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:42.349820 (+ 269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:42.349821 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:42.349821 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:42.349952 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:42.350301 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:42.350307 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:42.350468 (+ 161us) client_negotiation.cc:769] Sending connection context
0620 00:21:42.350565 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 00:21:42.350714 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":114,"threads_started":1}
I20260620 00:21:42.351528 17724 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37813
I20260620 00:21:42.351717 17724 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:42.351956 17724 heartbeater.cc:511] Master 127.0.98.62:37813 requested a full tablet report, sending...
I20260620 00:21:42.352435 17399 ts_manager.cc:194] Registered new tserver with Master: 797768b2f72b4c9cb5fba707b0564bff (127.0.98.2:35467)
I20260620 00:21:42.353001 17399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:56493
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:21:42.372933 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37385
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:42.460407 17729 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:42.460638 17729 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:42.460666 17729 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:42.462018 17729 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:42.462069 17729 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:42.462105 17729 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:21:42.463774 17729 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37385
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37813
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17729
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:42.464059 17729 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:42.464416 17729 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:42.469774 17737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:42.469774 17734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:42.469780 17735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:42.470059 17729 server_base.cc:1034] running on GCE node
I20260620 00:21:42.470266 17729 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:42.470474 17729 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:42.471626 17729 hybrid_clock.cc:648] HybridClock initialized: now 1781914902471610 us; error 29 us; skew 500 ppm
Jun 20 00:21:42 dist-test-slave-r81b krb5kdc[17354](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:42.474608 17729 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:21:42.475124 17729 webserver.cc:460] Webserver started at http://127.0.98.3:43903/ using document root <none> and password file <none>
I20260620 00:21:42.475384 17729 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:42.475455 17729 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:42.475637 17729 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:42.476862 17729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "76a797cf7d494ccca68960358c5c90b6"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "44327e5d45ca848ba35b2a6300f2cce4"
server_key_iv: "8cc66acc9a097f240e4e14b651f07e4a"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.477291 17729 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "76a797cf7d494ccca68960358c5c90b6"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "44327e5d45ca848ba35b2a6300f2cce4"
server_key_iv: "8cc66acc9a097f240e4e14b651f07e4a"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.479009 17729 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260620 00:21:42.480072 17744 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.480322 17729 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:42.480434 17729 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "76a797cf7d494ccca68960358c5c90b6"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "44327e5d45ca848ba35b2a6300f2cce4"
server_key_iv: "8cc66acc9a097f240e4e14b651f07e4a"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.480533 17729 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:42.495568 17729 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:42.498135 17729 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:42.498334 17729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:42.498594 17729 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:42.498977 17729 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:42.499038 17729 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.499080 17729 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:42.499110 17729 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.505545 17729 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35361
I20260620 00:21:42.505666 17857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35361 every 8 connection(s)
I20260620 00:21:42.506057 17729 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:21:42.509025 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 17729
I20260620 00:21:42.509133 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:21:42.509310 392 external_mini_cluster.cc:1428] Setting key 6e1854776fe0aea1897100492ad8e6ce
Jun 20 00:21:42 dist-test-slave-r81b krb5kdc[17354](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:42.521950 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.506861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35807 (local address 127.0.98.62:37813)
0620 00:21:42.506914 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.506915 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.507198 (+ 283us) server_negotiation.cc:407] Connection header received
0620 00:21:42.508196 (+ 998us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.508198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.508211 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.508269 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:42.508957 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.509674 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.510835 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.511046 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.511119 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.518427 (+ 7308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:42.518429 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:42.518429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:42.518439 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:42.520241 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:42.520733 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:42.520735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:42.520735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:42.520812 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:42.521176 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:42.521177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:42.521177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:42.521334 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:42.521418 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.521751 (+ 333us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.521852 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:21:42.522157 17860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.506839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37813 (local address 127.0.98.3:35807)
0620 00:21:42.507030 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:42.507094 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:42.508052 (+ 958us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:42.508347 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:42.508351 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:42.508629 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:42.508855 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.508860 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.509762 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.509763 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.510728 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.510731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.511146 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.511147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.511273 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.516645 (+ 5372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:42.516657 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:42.518270 (+ 1613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:42.520349 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:42.520358 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:42.520361 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:42.520629 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:42.520955 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:42.520956 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:42.520956 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:42.521071 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:42.521420 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:42.521428 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:42.521577 (+ 149us) client_negotiation.cc:769] Sending connection context
0620 00:21:42.521729 (+ 152us) client_negotiation.cc:240] Negotiation successful
0620 00:21:42.521913 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":92,"threads_started":1}
I20260620 00:21:42.522655 17858 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37813
I20260620 00:21:42.522778 17858 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:42.522971 17858 heartbeater.cc:511] Master 127.0.98.62:37813 requested a full tablet report, sending...
I20260620 00:21:42.523432 17399 ts_manager.cc:194] Registered new tserver with Master: 76a797cf7d494ccca68960358c5c90b6 (127.0.98.3:35361)
I20260620 00:21:42.523926 17399 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35807
I20260620 00:21:42.532774 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:21:42.541162 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.534258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57816 (local address 127.0.98.62:37813)
0620 00:21:42.534356 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.534358 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.534437 (+ 79us) server_negotiation.cc:407] Connection header received
0620 00:21:42.534479 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.534479 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.534489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.534545 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:42.535183 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.535737 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.536642 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.536833 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.536896 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.538242 (+ 1346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:42.538244 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:42.538244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:42.538251 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:42.539923 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:42.540328 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:42.540329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:42.540329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:42.540382 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:42.540644 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:42.540645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:42.540645 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:42.540838 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:42.540921 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.541029 (+ 108us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.541095 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 00:21:42.543057 17399 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57816:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:21:42.544863 17399 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:21:42.551702 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.547372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35467 (local address 127.0.0.1:38106)
0620 00:21:42.547774 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:42.547783 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:42.547839 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:42.548099 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:42.548102 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:42.548105 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:42.548226 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.548229 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.549293 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.549294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.550463 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.550465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.551361 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.551362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.551445 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.551461 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 00:21:42.551538 (+ 77us) client_negotiation.cc:240] Negotiation successful
0620 00:21:42.551594 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":92,"threads_started":1}
I20260620 00:21:42.551725 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.547587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38106 (local address 127.0.98.2:35467)
0620 00:21:42.547749 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.547754 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.547896 (+ 142us) server_negotiation.cc:407] Connection header received
0620 00:21:42.547947 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.547950 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.547967 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.548051 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:42.548317 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.549132 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.550546 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.551265 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.551342 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.551407 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.551570 (+ 163us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.551617 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":95,"threads_started":1}
I20260620 00:21:42.551949 17869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.547351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43961 (local address 127.0.0.1:50158)
0620 00:21:42.547465 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:42.547505 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:42.547578 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:42.548232 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:42.548233 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:42.548235 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:42.548300 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.548302 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.549755 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.549756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.550740 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.550742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.551705 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.551706 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.551807 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.551812 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:21:42.551846 (+ 34us) client_negotiation.cc:240] Negotiation successful
0620 00:21:42.551884 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":68,"threads_started":1}
I20260620 00:21:42.552531 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.547823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50158 (local address 127.0.98.1:43961)
0620 00:21:42.547977 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.547980 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.547989 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:21:42.548029 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.548031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.548045 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.548373 (+ 328us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:42.548463 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.549644 (+ 1181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.550819 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.551604 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.552302 (+ 698us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.552356 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.552378 (+ 22us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.552412 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260620 00:21:42.552538 17658 tablet_service.cc:1467] Processing CreateTablet for tablet d327745a856e4d4e8e3a4db5f3edc16d (DEFAULT_TABLE table=test-table [id=890163bde9164356916c162973f63dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:42.552961 17658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d327745a856e4d4e8e3a4db5f3edc16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:42.553107 17870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.547572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35361 (local address 127.0.0.1:39152)
0620 00:21:42.547979 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:42.547988 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:42.548097 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:42.548424 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:42.548424 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:42.548426 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:42.548460 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.548461 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.550810 (+ 2349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.550810 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.551590 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.551593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.552635 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.552636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.552922 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.552927 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:21:42.552994 (+ 67us) client_negotiation.cc:240] Negotiation successful
0620 00:21:42.553040 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":390,"thread_start_us":160,"threads_started":1}
I20260620 00:21:42.553140 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.547666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39152 (local address 127.0.98.3:35361)
0620 00:21:42.547871 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.547875 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.548065 (+ 190us) server_negotiation.cc:407] Connection header received
0620 00:21:42.548208 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.548211 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.548227 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.548293 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:42.548539 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.549637 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.551687 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.552451 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.552503 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.552557 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.553020 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.553064 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":126,"threads_started":1}
I20260620 00:21:42.553305 17524 tablet_service.cc:1467] Processing CreateTablet for tablet d327745a856e4d4e8e3a4db5f3edc16d (DEFAULT_TABLE table=test-table [id=890163bde9164356916c162973f63dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:42.553604 17524 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d327745a856e4d4e8e3a4db5f3edc16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:42.556260 17875 tablet_bootstrap.cc:492] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: Bootstrap starting.
I20260620 00:21:42.556723 17876 tablet_bootstrap.cc:492] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05: Bootstrap starting.
I20260620 00:21:42.557178 17875 tablet_bootstrap.cc:654] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:42.557741 17875 log.cc:824] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:42.557859 17792 tablet_service.cc:1467] Processing CreateTablet for tablet d327745a856e4d4e8e3a4db5f3edc16d (DEFAULT_TABLE table=test-table [id=890163bde9164356916c162973f63dbf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:42.558198 17792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d327745a856e4d4e8e3a4db5f3edc16d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:42.558537 17875 tablet_bootstrap.cc:492] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: No bootstrap required, opened a new log
I20260620 00:21:42.558597 17875 ts_tablet_manager.cc:1397] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:21:42.560357 17875 raft_consensus.cc:348] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } }
I20260620 00:21:42.560500 17875 raft_consensus.cc:374] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:42.560523 17875 raft_consensus.cc:729] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 797768b2f72b4c9cb5fba707b0564bff, State: Initialized, Role: FOLLOWER
I20260620 00:21:42.560650 17875 consensus_queue.cc:260] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } }
I20260620 00:21:42.560969 17875 ts_tablet_manager.cc:1428] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:21:42.560993 17724 heartbeater.cc:503] Master 127.0.98.62:37813 was elected leader, sending a full tablet report...
I20260620 00:21:42.561301 17879 tablet_bootstrap.cc:492] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: Bootstrap starting.
I20260620 00:21:42.561517 17876 tablet_bootstrap.cc:654] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:42.562027 17879 tablet_bootstrap.cc:654] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:42.562155 17876 log.cc:824] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:42.562621 17879 log.cc:824] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:42.563030 17876 tablet_bootstrap.cc:492] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05: No bootstrap required, opened a new log
I20260620 00:21:42.563107 17876 ts_tablet_manager.cc:1397] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260620 00:21:42.563459 17879 tablet_bootstrap.cc:492] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: No bootstrap required, opened a new log
I20260620 00:21:42.563539 17879 ts_tablet_manager.cc:1397] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:21:42.564878 17876 raft_consensus.cc:348] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } }
I20260620 00:21:42.565044 17876 raft_consensus.cc:374] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:42.565094 17876 raft_consensus.cc:729] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3cdf1d3f40d240d4b2c25901bff8ca05, State: Initialized, Role: FOLLOWER
I20260620 00:21:42.564980 17879 raft_consensus.cc:348] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } }
I20260620 00:21:42.565150 17879 raft_consensus.cc:374] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:42.565197 17879 raft_consensus.cc:729] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76a797cf7d494ccca68960358c5c90b6, State: Initialized, Role: FOLLOWER
I20260620 00:21:42.565217 17876 consensus_queue.cc:260] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } }
I20260620 00:21:42.565339 17879 consensus_queue.cc:260] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } }
I20260620 00:21:42.565506 17876 ts_tablet_manager.cc:1428] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260620 00:21:42.565527 17590 heartbeater.cc:503] Master 127.0.98.62:37813 was elected leader, sending a full tablet report...
I20260620 00:21:42.565630 17879 ts_tablet_manager.cc:1428] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:21:42.565644 17858 heartbeater.cc:503] Master 127.0.98.62:37813 was elected leader, sending a full tablet report...
W20260620 00:21:42.589962 17725 tablet.cc:2369] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:42.665088 17882 raft_consensus.cc:482] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:42.665287 17882 raft_consensus.cc:504] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } }
I20260620 00:21:42.665673 17882 leader_election.cc:288] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 797768b2f72b4c9cb5fba707b0564bff (127.0.98.2:35467), 76a797cf7d494ccca68960358c5c90b6 (127.0.98.3:35361)
I20260620 00:21:42.670115 17871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.666031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41651 (local address 127.0.98.2:35467)
0620 00:21:42.666079 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.666082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.666141 (+ 59us) server_negotiation.cc:407] Connection header received
0620 00:21:42.666171 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.666172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.666186 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.666240 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:42.666424 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.667235 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.668968 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.669666 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.669748 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.669764 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.669992 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.670040 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:21:42.670123 17592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.665943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35467 (local address 127.0.98.1:41651)
0620 00:21:42.666014 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:42.666028 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:42.666093 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:42.666261 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:42.666262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:42.666264 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:42.666338 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.666340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.667357 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.667358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.668823 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.668826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.669768 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.669769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.669868 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.669870 (+ 2us) client_negotiation.cc:769] Sending connection context
0620 00:21:42.669943 (+ 73us) client_negotiation.cc:240] Negotiation successful
0620 00:21:42.670002 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47,"mutex_wait_us":119}
I20260620 00:21:42.670115 17873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.666272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:50513 (local address 127.0.98.3:35361)
0620 00:21:42.666320 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.666322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.666329 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:21:42.666361 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.666361 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.666371 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.666449 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:42.666703 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.667385 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.668966 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.669649 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.669754 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.669764 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.669988 (+ 224us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.670040 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:21:42.670127 17884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.666055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35361 (local address 127.0.98.1:50513)
0620 00:21:42.666195 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:42.666205 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:42.666255 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:42.666514 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:42.666515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:42.666517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:42.666603 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.666605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.667476 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.667476 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.668823 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:42.668826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.669768 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.669769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:42.669864 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.669869 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:21:42.669941 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:21:42.670002 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":41,"thread_start_us":86,"threads_started":1}
I20260620 00:21:42.670560 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d327745a856e4d4e8e3a4db5f3edc16d" candidate_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797768b2f72b4c9cb5fba707b0564bff" is_pre_election: true
I20260620 00:21:42.670722 17678 raft_consensus.cc:2454] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cdf1d3f40d240d4b2c25901bff8ca05 in term 0.
I20260620 00:21:42.670979 17478 leader_election.cc:302] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cdf1d3f40d240d4b2c25901bff8ca05, 797768b2f72b4c9cb5fba707b0564bff; no voters:
I20260620 00:21:42.671132 17882 raft_consensus.cc:2781] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:21:42.671077 17812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d327745a856e4d4e8e3a4db5f3edc16d" candidate_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a797cf7d494ccca68960358c5c90b6" is_pre_election: true
I20260620 00:21:42.671197 17882 raft_consensus.cc:482] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:21:42.671216 17882 raft_consensus.cc:3037] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:42.671279 17812 raft_consensus.cc:2454] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3cdf1d3f40d240d4b2c25901bff8ca05 in term 0.
I20260620 00:21:42.672063 17882 raft_consensus.cc:504] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } }
I20260620 00:21:42.672233 17882 leader_election.cc:288] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [CANDIDATE]: Term 1 election: Requested vote from peers 797768b2f72b4c9cb5fba707b0564bff (127.0.98.2:35467), 76a797cf7d494ccca68960358c5c90b6 (127.0.98.3:35361)
I20260620 00:21:42.672392 17678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d327745a856e4d4e8e3a4db5f3edc16d" candidate_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "797768b2f72b4c9cb5fba707b0564bff"
I20260620 00:21:42.672416 17812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d327745a856e4d4e8e3a4db5f3edc16d" candidate_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76a797cf7d494ccca68960358c5c90b6"
I20260620 00:21:42.672479 17678 raft_consensus.cc:3037] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:42.672501 17812 raft_consensus.cc:3037] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:42.673353 17678 raft_consensus.cc:2454] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cdf1d3f40d240d4b2c25901bff8ca05 in term 1.
I20260620 00:21:42.673465 17812 raft_consensus.cc:2454] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3cdf1d3f40d240d4b2c25901bff8ca05 in term 1.
I20260620 00:21:42.673672 17478 leader_election.cc:302] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3cdf1d3f40d240d4b2c25901bff8ca05, 797768b2f72b4c9cb5fba707b0564bff; no voters:
I20260620 00:21:42.673774 17882 raft_consensus.cc:2781] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:42.673964 17882 raft_consensus.cc:686] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 1 LEADER]: Becoming Leader. State: Replica: 3cdf1d3f40d240d4b2c25901bff8ca05, State: Running, Role: LEADER
I20260620 00:21:42.674088 17882 consensus_queue.cc:237] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } }
I20260620 00:21:42.675076 17398 catalog_manager.cc:5495] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3cdf1d3f40d240d4b2c25901bff8ca05 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3cdf1d3f40d240d4b2c25901bff8ca05" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43961 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } health_report { overall_health: UNKNOWN } } }
W20260620 00:21:42.679975 17591 tablet.cc:2369] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:42.699520 17874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.696899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50160 (local address 127.0.98.1:43961)
0620 00:21:42.696942 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.696944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.696958 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:21:42.697022 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.697023 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.697043 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.697156 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:21:42.697334 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.697857 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.698668 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.698863 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.698944 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.699133 (+ 189us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:21:42.699246 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:21:42.699350 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.699369 (+ 19us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.699393 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:21:42.701601 17812 raft_consensus.cc:1261] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [term 1 FOLLOWER]: Refusing update from remote peer 3cdf1d3f40d240d4b2c25901bff8ca05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:42.701601 17678 raft_consensus.cc:1261] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [term 1 FOLLOWER]: Refusing update from remote peer 3cdf1d3f40d240d4b2c25901bff8ca05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:42.702031 17882 consensus_queue.cc:1035] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "76a797cf7d494ccca68960358c5c90b6" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:42.702284 17885 consensus_queue.cc:1035] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "797768b2f72b4c9cb5fba707b0564bff" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:42.705291 17889 mvcc.cc:205] Tried to move back new op lower bound from 7298723441462677504 to 7298723441354055680. Current Snapshot: MvccSnapshot[applied={T|T < 7298723441462677504}]
I20260620 00:21:42.708613 17399 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:42.708748 17399 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:42.709648 17399 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a50244bf8ad747f4983c523a7fe77152: Sending DeleteTablet for 3 replicas of tablet d327745a856e4d4e8e3a4db5f3edc16d
I20260620 00:21:42.710114 17658 tablet_service.cc:1514] Processing DeleteTablet for tablet d327745a856e4d4e8e3a4db5f3edc16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38106
I20260620 00:21:42.710100 17792 tablet_service.cc:1514] Processing DeleteTablet for tablet d327745a856e4d4e8e3a4db5f3edc16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39152
I20260620 00:21:42.710124 17524 tablet_service.cc:1514] Processing DeleteTablet for tablet d327745a856e4d4e8e3a4db5f3edc16d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:42 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50158
I20260620 00:21:42.710376 17900 tablet_replica.cc:330] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05: stopping tablet replica
I20260620 00:21:42.710485 17900 raft_consensus.cc:2229] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:21:42.710633 17900 raft_consensus.cc:2258] T d327745a856e4d4e8e3a4db5f3edc16d P 3cdf1d3f40d240d4b2c25901bff8ca05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:42.710762 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 17461
I20260620 00:21:42.711467 17899 tablet_replica.cc:330] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: stopping tablet replica
I20260620 00:21:42.711611 17899 raft_consensus.cc:2229] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:42.711751 17899 raft_consensus.cc:2258] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:42.712065 17898 tablet_replica.cc:330] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: stopping tablet replica
I20260620 00:21:42.712250 17898 raft_consensus.cc:2229] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:42.712249 17899 ts_tablet_manager.cc:1905] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:42.712385 17898 raft_consensus.cc:2258] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:42.712769 17898 ts_tablet_manager.cc:1905] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:42.714493 17899 ts_tablet_manager.cc:1918] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:42.714478 17898 ts_tablet_manager.cc:1918] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:42.714572 17898 log.cc:1196] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/d327745a856e4d4e8e3a4db5f3edc16d
I20260620 00:21:42.714587 17899 log.cc:1196] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/d327745a856e4d4e8e3a4db5f3edc16d
I20260620 00:21:42.714864 17898 ts_tablet_manager.cc:1939] T d327745a856e4d4e8e3a4db5f3edc16d P 797768b2f72b4c9cb5fba707b0564bff: Deleting consensus metadata
I20260620 00:21:42.714888 17899 ts_tablet_manager.cc:1939] T d327745a856e4d4e8e3a4db5f3edc16d P 76a797cf7d494ccca68960358c5c90b6: Deleting consensus metadata
I20260620 00:21:42.715342 17384 catalog_manager.cc:4841] TS 797768b2f72b4c9cb5fba707b0564bff (127.0.98.2:35467): tablet d327745a856e4d4e8e3a4db5f3edc16d (table test-table [id=890163bde9164356916c162973f63dbf]) successfully deleted
I20260620 00:21:42.715456 17384 catalog_manager.cc:4841] TS 76a797cf7d494ccca68960358c5c90b6 (127.0.98.3:35361): tablet d327745a856e4d4e8e3a4db5f3edc16d (table test-table [id=890163bde9164356916c162973f63dbf]) successfully deleted
W20260620 00:21:42.718381 17383 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43961 (error 108)
I20260620 00:21:42.718611 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 17595
I20260620 00:21:42.719843 17872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.719703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43961 (local address 127.0.0.1:50166)
0620 00:21:42.719765 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:42.719811 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 00:21:42.720009 17383 catalog_manager.cc:4584] TS 3cdf1d3f40d240d4b2c25901bff8ca05 (127.0.98.1:43961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d327745a856e4d4e8e3a4db5f3edc16d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43961: connect: Connection refused (error 111)
I20260620 00:21:42.723757 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 17729
I20260620 00:21:42.728696 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 17367
2026-06-20T00:21:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2073 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-06-20T00:21:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:42Z Disabled control of system clock
I20260620 00:21:42.756964 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33359
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33359
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:21:42.845464 17905 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:42.845849 17905 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:42.845894 17905 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:42.847429 17905 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:42.847492 17905 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:42.847506 17905 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:42.847520 17905 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:42.847532 17905 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:42.849107 17905 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33359
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33359
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17905
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:42.849380 17905 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:42.849669 17905 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:42.855077 17911 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:42.855050 17913 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:42.855159 17910 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:42.855466 17905 server_base.cc:1034] running on GCE node
I20260620 00:21:42.855741 17905 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:42.856045 17905 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:42.857218 17905 hybrid_clock.cc:648] HybridClock initialized: now 1781914902857198 us; error 34 us; skew 500 ppm
I20260620 00:21:42.858810 17905 webserver.cc:460] Webserver started at http://127.0.98.62:41879/ using document root <none> and password file <none>
I20260620 00:21:42.859127 17905 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:42.859186 17905 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:42.859287 17905 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:42.860499 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "476ca4ad5cb94881be883b83d4c8a463"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "3170c4de8323311f96f81a1063e1da61"
server_key_iv: "b48c7119ec8f5b033f1edc7f91769e37"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.860826 17905 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "476ca4ad5cb94881be883b83d4c8a463"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "3170c4de8323311f96f81a1063e1da61"
server_key_iv: "b48c7119ec8f5b033f1edc7f91769e37"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.862196 17905 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:21:42.863094 17919 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:42.863349 17905 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:42.863427 17905 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "476ca4ad5cb94881be883b83d4c8a463"
format_stamp: "Formatted at 2026-06-20 00:21:42 on dist-test-slave-r81b"
server_key: "3170c4de8323311f96f81a1063e1da61"
server_key_iv: "b48c7119ec8f5b033f1edc7f91769e37"
server_key_version: "encryptionkey@0"
I20260620 00:21:42.863500 17905 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:42.875579 17905 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:42.875897 17905 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:42.876011 17905 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:42.880069 17905 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33359
I20260620 00:21:42.880118 17971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33359 every 8 connection(s)
I20260620 00:21:42.880594 17905 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:42.881610 17972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:42.882043 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 17905
I20260620 00:21:42.882169 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:42.882408 392 external_mini_cluster.cc:1428] Setting key 1b5aeef4a9091b35bcd2303a49cbf04b
I20260620 00:21:42.884646 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463: Bootstrap starting.
I20260620 00:21:42.885300 17972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:42.885612 17972 log.cc:824] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:42.886543 17972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463: No bootstrap required, opened a new log
I20260620 00:21:42.888984 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:42.883014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56442 (local address 127.0.98.62:33359)
0620 00:21:42.883256 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:42.883272 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:42.883299 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:21:42.883795 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:42.883804 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:42.883873 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:42.884035 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:42.884263 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.885224 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.886190 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:42.886829 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:42.886969 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:42.887815 (+ 846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:42.887821 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:42.887821 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:42.888042 (+ 221us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:42.888091 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:42.888107 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:42.888210 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:42.888355 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 00:21:42.888452 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":86,"threads_started":1}
I20260620 00:21:42.888839 17972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476ca4ad5cb94881be883b83d4c8a463" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33359 } }
I20260620 00:21:42.889057 17972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:42.889096 17972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 476ca4ad5cb94881be883b83d4c8a463, State: Initialized, Role: FOLLOWER
I20260620 00:21:42.889225 17972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476ca4ad5cb94881be883b83d4c8a463" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33359 } }
I20260620 00:21:42.889321 17972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:42.889351 17972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:42.889400 17972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:42.890017 17972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476ca4ad5cb94881be883b83d4c8a463" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33359 } }
I20260620 00:21:42.890147 17972 leader_election.cc:302] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 476ca4ad5cb94881be883b83d4c8a463; no voters:
I20260620 00:21:42.890367 17972 leader_election.cc:288] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:42.890416 17977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:42.890587 17977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [term 1 LEADER]: Becoming Leader. State: Replica: 476ca4ad5cb94881be883b83d4c8a463, State: Running, Role: LEADER
I20260620 00:21:42.890708 17972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:42.890733 17977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476ca4ad5cb94881be883b83d4c8a463" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33359 } }
I20260620 00:21:42.891177 17978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "476ca4ad5cb94881be883b83d4c8a463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476ca4ad5cb94881be883b83d4c8a463" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33359 } } }
I20260620 00:21:42.891211 17979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 476ca4ad5cb94881be883b83d4c8a463. Latest consensus state: current_term: 1 leader_uuid: "476ca4ad5cb94881be883b83d4c8a463" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "476ca4ad5cb94881be883b83d4c8a463" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33359 } } }
I20260620 00:21:42.891275 17978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:42.891284 17979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:42.891466 17986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:42.891971 17986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:42.893700 17986 catalog_manager.cc:1347] Generated new cluster ID: 1854864014454ea09c02f8ccace48a42
I20260620 00:21:42.893766 17986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:42.907264 17986 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:42.907876 17986 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:42.913002 17986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463: Generated new TSK 0
I20260620 00:21:42.913228 17986 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:42.923796 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33359
--builtin_ntp_servers=127.0.98.20:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:21:43.014103 17996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:43.014370 17996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:43.014397 17996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:43.015799 17996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:43.015853 17996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:43.015887 17996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:21:43.017627 17996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:43.017925 17996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:43.018308 17996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:21:43.024982 17996 server_base.cc:1034] running on GCE node
W20260620 00:21:43.024905 18002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:43.024899 18001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:43.024928 18004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:43.025393 17996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:43.025724 17996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:43.026943 17996 hybrid_clock.cc:648] HybridClock initialized: now 1781914903026913 us; error 45 us; skew 500 ppm
I20260620 00:21:43.028637 17996 webserver.cc:460] Webserver started at http://127.0.98.1:36449/ using document root <none> and password file <none>
I20260620 00:21:43.028963 17996 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:43.029058 17996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:43.029207 17996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:43.030370 17996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "8f8d1beaf9c143349e01ea49e363c15f"
format_stamp: "Formatted at 2026-06-20 00:21:43 on dist-test-slave-r81b"
server_key: "311540eb4869462e2cf4000f9e00e90e"
server_key_iv: "4e388dc5eed0d6c22924c9b771506c62"
server_key_version: "encryptionkey@0"
I20260620 00:21:43.030764 17996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8f8d1beaf9c143349e01ea49e363c15f"
format_stamp: "Formatted at 2026-06-20 00:21:43 on dist-test-slave-r81b"
server_key: "311540eb4869462e2cf4000f9e00e90e"
server_key_iv: "4e388dc5eed0d6c22924c9b771506c62"
server_key_version: "encryptionkey@0"
I20260620 00:21:43.032248 17996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:21:43.033159 18010 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:43.033579 17996 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:43.033795 17996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "8f8d1beaf9c143349e01ea49e363c15f"
format_stamp: "Formatted at 2026-06-20 00:21:43 on dist-test-slave-r81b"
server_key: "311540eb4869462e2cf4000f9e00e90e"
server_key_iv: "4e388dc5eed0d6c22924c9b771506c62"
server_key_version: "encryptionkey@0"
I20260620 00:21:43.033880 17996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:43.047637 17996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:43.047943 17996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:43.048054 17996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:43.048339 17996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:43.048656 17996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:43.048691 17996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:43.048713 17996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:43.048727 17996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:43.055070 17996 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35101
I20260620 00:21:43.055119 18123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35101 every 8 connection(s)
I20260620 00:21:43.055461 17996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:21:43.059751 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 17996
I20260620 00:21:43.059850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:21:43.060104 392 external_mini_cluster.cc:1428] Setting key 1b3f6ac162436c0406de2a25b42ac324
I20260620 00:21:43.061444 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33359
--builtin_ntp_servers=127.0.98.20:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:21:43.062140 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.056367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40259 (local address 127.0.98.62:33359)
0620 00:21:43.056462 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.056464 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.056803 (+ 339us) server_negotiation.cc:407] Connection header received
0620 00:21:43.057426 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.057428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.057446 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.057513 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:43.058306 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.058901 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.060129 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.060460 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.060532 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.061810 (+ 1278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:43.061812 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:43.061812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:43.061818 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:43.061835 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:43.061840 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:43.061898 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.062013 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.062050 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260620 00:21:43.062436 18126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.056380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33359 (local address 127.0.98.1:40259)
0620 00:21:43.056628 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:43.056696 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:43.057300 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:43.057595 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:43.057600 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:43.057609 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:43.058195 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.058200 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.059024 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.059025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.060027 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.060030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.060541 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.060543 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.060709 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.061456 (+ 747us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:21:43.061457 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:43.061492 (+ 35us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:21:43.061519 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:43.061911 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:43.061917 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:43.061921 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:43.062018 (+ 97us) client_negotiation.cc:240] Negotiation successful
0620 00:21:43.062141 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":100,"threads_started":1}
I20260620 00:21:43.063050 18124 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33359
I20260620 00:21:43.063169 18124 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:43.063410 18124 heartbeater.cc:511] Master 127.0.98.62:33359 requested a full tablet report, sending...
I20260620 00:21:43.064528 17936 ts_manager.cc:194] Registered new tserver with Master: 8f8d1beaf9c143349e01ea49e363c15f (127.0.98.1:35101)
I20260620 00:21:43.065629 17936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40259
W20260620 00:21:43.153803 18127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:43.154102 18127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:43.154300 18127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:43.155714 18127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:43.155800 18127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:43.155848 18127 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:21:43.157444 18127 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:43.157758 18127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:43.158114 18127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:43.163779 18132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:43.163820 18135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:43.164283 18133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:43.164366 18127 server_base.cc:1034] running on GCE node
I20260620 00:21:43.164621 18127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:43.164894 18127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:43.166090 18127 hybrid_clock.cc:648] HybridClock initialized: now 1781914903166079 us; error 45 us; skew 500 ppm
I20260620 00:21:43.167832 18127 webserver.cc:460] Webserver started at http://127.0.98.2:40259/ using document root <none> and password file <none>
I20260620 00:21:43.168126 18127 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:43.168272 18127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:43.168447 18127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:43.169624 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "232876eee09547499947228721aa2ed7"
format_stamp: "Formatted at 2026-06-20 00:21:43 on dist-test-slave-r81b"
server_key: "cb6ded117ae6e2d4ffe01369196fcd27"
server_key_iv: "90a8c1fd2c0e1d12936371b28ef39ee1"
server_key_version: "encryptionkey@0"
I20260620 00:21:43.170022 18127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "232876eee09547499947228721aa2ed7"
format_stamp: "Formatted at 2026-06-20 00:21:43 on dist-test-slave-r81b"
server_key: "cb6ded117ae6e2d4ffe01369196fcd27"
server_key_iv: "90a8c1fd2c0e1d12936371b28ef39ee1"
server_key_version: "encryptionkey@0"
I20260620 00:21:43.171438 18127 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:21:43.172387 18141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:43.172603 18127 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:43.172719 18127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "232876eee09547499947228721aa2ed7"
format_stamp: "Formatted at 2026-06-20 00:21:43 on dist-test-slave-r81b"
server_key: "cb6ded117ae6e2d4ffe01369196fcd27"
server_key_iv: "90a8c1fd2c0e1d12936371b28ef39ee1"
server_key_version: "encryptionkey@0"
I20260620 00:21:43.172820 18127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:43.201566 18127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:43.201958 18127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:43.202139 18127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:43.202404 18127 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:43.202767 18127 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:43.202824 18127 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:43.202868 18127 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:43.202898 18127 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:43.209400 18127 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33983
I20260620 00:21:43.209460 18254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33983 every 8 connection(s)
I20260620 00:21:43.209869 18127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:21:43.215799 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.210677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60525 (local address 127.0.98.62:33359)
0620 00:21:43.210748 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.210750 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.210982 (+ 232us) server_negotiation.cc:407] Connection header received
0620 00:21:43.211540 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.211541 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.211552 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.211616 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:43.212474 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.213023 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.214248 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.214455 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.214520 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.215488 (+ 968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:43.215489 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:43.215489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:43.215495 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:43.215510 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:43.215514 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:43.215568 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.215703 (+ 135us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.215731 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260620 00:21:43.216037 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.210649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33359 (local address 127.0.98.2:60525)
0620 00:21:43.210845 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:43.210880 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:43.211396 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:43.211762 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:43.211766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:43.211776 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:43.212348 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.212353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.213151 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.213152 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.214125 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.214128 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.214551 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.214552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.214684 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.215305 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:21:43.215306 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:43.215342 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:21:43.215370 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:43.215581 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:43.215586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:43.215589 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:43.215652 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 00:21:43.215753 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":102,"threads_started":1}
I20260620 00:21:43.216702 18255 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33359
I20260620 00:21:43.216871 18255 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:43.217089 18255 heartbeater.cc:511] Master 127.0.98.62:33359 requested a full tablet report, sending...
I20260620 00:21:43.217593 17936 ts_manager.cc:194] Registered new tserver with Master: 232876eee09547499947228721aa2ed7 (127.0.98.2:33983)
I20260620 00:21:43.218021 17936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:60525
I20260620 00:21:43.220160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 18127
I20260620 00:21:43.220299 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:21:43.220520 392 external_mini_cluster.cc:1428] Setting key e147c73b50ccc8fed5ca39433345e70d
I20260620 00:21:43.221634 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33359
--builtin_ntp_servers=127.0.98.20:43667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:21:43.307266 18258 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:43.307514 18258 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:43.307539 18258 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:43.308897 18258 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:43.308951 18258 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:43.308980 18258 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:21:43.310523 18258 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33359
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18258
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:43.310820 18258 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:43.311182 18258 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:43.316502 18263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:43.316699 18266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:43.316742 18264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:43.316910 18258 server_base.cc:1034] running on GCE node
I20260620 00:21:43.317130 18258 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:43.317375 18258 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:43.318524 18258 hybrid_clock.cc:648] HybridClock initialized: now 1781914903318514 us; error 34 us; skew 500 ppm
I20260620 00:21:43.319980 18258 webserver.cc:460] Webserver started at http://127.0.98.3:43939/ using document root <none> and password file <none>
I20260620 00:21:43.320320 18258 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:43.320398 18258 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:43.320556 18258 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:43.321674 18258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "565a304ea03b4b469d1fdee368fd2334"
format_stamp: "Formatted at 2026-06-20 00:21:43 on dist-test-slave-r81b"
server_key: "59a1d67bf4b6ded238956902f70dd66a"
server_key_iv: "8cc39018a7466fbf260d794acd222d76"
server_key_version: "encryptionkey@0"
I20260620 00:21:43.322063 18258 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "565a304ea03b4b469d1fdee368fd2334"
format_stamp: "Formatted at 2026-06-20 00:21:43 on dist-test-slave-r81b"
server_key: "59a1d67bf4b6ded238956902f70dd66a"
server_key_iv: "8cc39018a7466fbf260d794acd222d76"
server_key_version: "encryptionkey@0"
I20260620 00:21:43.323488 18258 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:21:43.324463 18272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:43.324863 18258 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:43.324987 18258 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "565a304ea03b4b469d1fdee368fd2334"
format_stamp: "Formatted at 2026-06-20 00:21:43 on dist-test-slave-r81b"
server_key: "59a1d67bf4b6ded238956902f70dd66a"
server_key_iv: "8cc39018a7466fbf260d794acd222d76"
server_key_version: "encryptionkey@0"
I20260620 00:21:43.325078 18258 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:43.340608 18258 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:43.341001 18258 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:43.341172 18258 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:43.341418 18258 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:43.341770 18258 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:43.341828 18258 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:43.341871 18258 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:43.341904 18258 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:43.347867 18258 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35735
I20260620 00:21:43.347941 18385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35735 every 8 connection(s)
I20260620 00:21:43.348367 18258 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:21:43.354063 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.349192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33975 (local address 127.0.98.62:33359)
0620 00:21:43.349308 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.349311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.349532 (+ 221us) server_negotiation.cc:407] Connection header received
0620 00:21:43.350127 (+ 595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.350128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.350142 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.350197 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:43.350914 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.351435 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.352484 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.352705 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.352783 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.353711 (+ 928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:43.353713 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:43.353713 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:43.353719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:43.353736 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:43.353743 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:43.353810 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.353910 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.353941 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260620 00:21:43.354281 18388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.349175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33359 (local address 127.0.98.3:33975)
0620 00:21:43.349404 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:43.349439 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:43.350008 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:43.350273 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:43.350278 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:43.350288 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:43.350812 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.350817 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.351538 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.351539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.352386 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.352389 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.352803 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.352804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.352943 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.353500 (+ 557us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:21:43.353500 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:43.353531 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:21:43.353556 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:43.353811 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:43.353818 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:43.353821 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:43.353914 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:21:43.354032 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
I20260620 00:21:43.354841 18386 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33359
I20260620 00:21:43.354934 18386 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:43.355083 18386 heartbeater.cc:511] Master 127.0.98.62:33359 requested a full tablet report, sending...
I20260620 00:21:43.355480 17936 ts_manager.cc:194] Registered new tserver with Master: 565a304ea03b4b469d1fdee368fd2334 (127.0.98.3:35735)
I20260620 00:21:43.355872 17936 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:33975
I20260620 00:21:43.357457 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 18258
I20260620 00:21:43.357550 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:21:43.357775 392 external_mini_cluster.cc:1428] Setting key 738bfc51de9cf4f812bf4328dd27fc40
I20260620 00:21:43.358816 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:21:43.363175 17975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.360242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56448 (local address 127.0.98.62:33359)
0620 00:21:43.360306 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.360308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.360437 (+ 129us) server_negotiation.cc:407] Connection header received
0620 00:21:43.360477 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.360478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.360487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.360535 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:43.360723 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.361189 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.361944 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.362145 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.362196 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.362855 (+ 659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:43.362857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:43.362857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:43.362862 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:43.362881 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:43.362885 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:43.362933 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.363042 (+ 109us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.363070 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260620 00:21:43.364972 17936 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:21:43.366732 17936 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:21:43.373950 18397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.369429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35735 (local address 127.0.0.1:55466)
0620 00:21:43.369814 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:43.369864 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:43.369941 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:43.370317 (+ 376us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:43.370320 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:43.370323 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:43.370411 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.370414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.371787 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.371788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.372860 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.372862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.373739 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.373740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.373795 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.373798 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:43.373842 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:21:43.373871 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":6,"thread_start_us":65,"threads_started":1}
I20260620 00:21:43.373988 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.369532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55466 (local address 127.0.98.3:35735)
0620 00:21:43.370131 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.370135 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.370144 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:21:43.370195 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.370198 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.370212 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.370493 (+ 281us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:43.370558 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.371702 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.372943 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.373629 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.373734 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.373824 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.373853 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.373893 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":107,"threads_started":1}
I20260620 00:21:43.374313 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.369768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36694 (local address 127.0.98.1:35101)
0620 00:21:43.369984 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.369990 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.370147 (+ 157us) server_negotiation.cc:407] Connection header received
0620 00:21:43.370191 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.370193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.370209 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.370280 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:43.370500 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.371256 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.372720 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.373480 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.373582 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.373686 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.373991 (+ 305us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.374216 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":101,"threads_started":1}
I20260620 00:21:43.374544 18394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.369429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33983 (local address 127.0.0.1:58070)
0620 00:21:43.369842 (+ 413us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:43.369864 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:43.369945 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:43.370327 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:43.370328 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:43.370329 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:43.370411 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.370414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.372194 (+ 1780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.372195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.373164 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.373167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.374297 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.374299 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.374349 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.374353 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:43.374410 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:21:43.374464 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":7,"thread_start_us":66,"threads_started":1}
I20260620 00:21:43.374994 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.369671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58070 (local address 127.0.98.2:33983)
0620 00:21:43.370092 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.370098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.370111 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:21:43.370155 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.370157 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.370174 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.370252 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:43.370495 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.371689 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.373265 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.374001 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.374686 (+ 685us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.374841 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.374870 (+ 29us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.374918 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":378,"thread_start_us":83,"threads_started":1}
I20260620 00:21:43.375003 18320 tablet_service.cc:1467] Processing CreateTablet for tablet 7924463faa11468b9086a192ae23c01a (DEFAULT_TABLE table=test-table [id=2149375b58464597b789f39a9dc29e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:43.375437 18320 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7924463faa11468b9086a192ae23c01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:43.375710 18189 tablet_service.cc:1467] Processing CreateTablet for tablet 7924463faa11468b9086a192ae23c01a (DEFAULT_TABLE table=test-table [id=2149375b58464597b789f39a9dc29e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:43.376073 18189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7924463faa11468b9086a192ae23c01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:43.373950 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.369638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35101 (local address 127.0.0.1:36694)
0620 00:21:43.370044 (+ 406us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:43.370052 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:43.370094 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:43.370349 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:43.370350 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:43.370352 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:43.370413 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.370414 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.371357 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.371358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.372616 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.372619 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.373587 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.373588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.373679 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.373695 (+ 16us) client_negotiation.cc:769] Sending connection context
0620 00:21:43.373767 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:21:43.373834 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"mutex_wait_us":8,"thread_start_us":95,"threads_started":1}
I20260620 00:21:43.377151 18058 tablet_service.cc:1467] Processing CreateTablet for tablet 7924463faa11468b9086a192ae23c01a (DEFAULT_TABLE table=test-table [id=2149375b58464597b789f39a9dc29e98]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:43.377491 18058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7924463faa11468b9086a192ae23c01a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:43.378463 18400 tablet_bootstrap.cc:492] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: Bootstrap starting.
I20260620 00:21:43.379138 18400 tablet_bootstrap.cc:654] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:43.379485 18400 log.cc:824] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:43.379509 18401 tablet_bootstrap.cc:492] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: Bootstrap starting.
I20260620 00:21:43.380134 18401 tablet_bootstrap.cc:654] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:43.380332 18400 tablet_bootstrap.cc:492] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: No bootstrap required, opened a new log
I20260620 00:21:43.380391 18400 ts_tablet_manager.cc:1397] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:21:43.380497 18401 log.cc:824] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:43.380825 18403 tablet_bootstrap.cc:492] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f: Bootstrap starting.
I20260620 00:21:43.381477 18403 tablet_bootstrap.cc:654] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:43.381768 18403 log.cc:824] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:43.382460 18403 tablet_bootstrap.cc:492] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f: No bootstrap required, opened a new log
I20260620 00:21:43.382550 18403 ts_tablet_manager.cc:1397] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:21:43.382649 18400 raft_consensus.cc:348] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } }
I20260620 00:21:43.382802 18400 raft_consensus.cc:374] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:43.382824 18400 raft_consensus.cc:729] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565a304ea03b4b469d1fdee368fd2334, State: Initialized, Role: FOLLOWER
I20260620 00:21:43.382961 18400 consensus_queue.cc:260] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } }
I20260620 00:21:43.383221 18400 ts_tablet_manager.cc:1428] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:21:43.383307 18386 heartbeater.cc:503] Master 127.0.98.62:33359 was elected leader, sending a full tablet report...
I20260620 00:21:43.383916 18401 tablet_bootstrap.cc:492] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: No bootstrap required, opened a new log
I20260620 00:21:43.384033 18401 ts_tablet_manager.cc:1397] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260620 00:21:43.385222 18403 raft_consensus.cc:348] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } }
I20260620 00:21:43.385399 18403 raft_consensus.cc:374] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:43.385437 18403 raft_consensus.cc:729] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f8d1beaf9c143349e01ea49e363c15f, State: Initialized, Role: FOLLOWER
I20260620 00:21:43.385562 18403 consensus_queue.cc:260] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } }
I20260620 00:21:43.385828 18403 ts_tablet_manager.cc:1428] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:21:43.385896 18124 heartbeater.cc:503] Master 127.0.98.62:33359 was elected leader, sending a full tablet report...
I20260620 00:21:43.386613 18401 raft_consensus.cc:348] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } }
I20260620 00:21:43.386765 18401 raft_consensus.cc:374] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:43.386809 18401 raft_consensus.cc:729] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 232876eee09547499947228721aa2ed7, State: Initialized, Role: FOLLOWER
I20260620 00:21:43.386927 18401 consensus_queue.cc:260] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } }
I20260620 00:21:43.387171 18401 ts_tablet_manager.cc:1428] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:21:43.387250 18255 heartbeater.cc:503] Master 127.0.98.62:33359 was elected leader, sending a full tablet report...
W20260620 00:21:43.460465 18256 tablet.cc:2369] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:43.552802 18408 raft_consensus.cc:482] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:43.552945 18408 raft_consensus.cc:504] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } }
I20260620 00:21:43.553531 18408 leader_election.cc:288] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f8d1beaf9c143349e01ea49e363c15f (127.0.98.1:35101), 565a304ea03b4b469d1fdee368fd2334 (127.0.98.3:35735)
W20260620 00:21:43.556296 18125 tablet.cc:2369] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:43.558048 18399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.553859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38389 (local address 127.0.98.1:35101)
0620 00:21:43.553903 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.553905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.554069 (+ 164us) server_negotiation.cc:407] Connection header received
0620 00:21:43.554108 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.554109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.554122 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.554182 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:43.554403 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.555188 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.556904 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.557561 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.557633 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.557644 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.557890 (+ 246us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.557953 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:21:43.558099 18257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.553766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35735 (local address 127.0.98.2:52867)
0620 00:21:43.553867 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:43.553881 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:43.553933 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:43.554115 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:43.554117 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:43.554119 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:43.554251 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.554263 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.555323 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.555324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.556728 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.556733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.557748 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.557749 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.557886 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.557891 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:21:43.557943 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:21:43.558006 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":8}
I20260620 00:21:43.558194 18396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.553892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52867 (local address 127.0.98.3:35735)
0620 00:21:43.553941 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.553943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.553956 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:21:43.553998 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.553998 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.554006 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.554098 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:43.554404 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.555201 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.556839 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.557496 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.557595 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.557619 (+ 24us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.557997 (+ 378us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.558077 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:21:43.558555 18409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.553778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35101 (local address 127.0.98.2:38389)
0620 00:21:43.553985 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:43.553995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:43.554048 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:43.554193 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:43.554194 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:43.554195 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:43.554251 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.554253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.555323 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.555324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.556778 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:43.556779 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.557670 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.557671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:43.557762 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.557768 (+ 6us) client_negotiation.cc:769] Sending connection context
0620 00:21:43.558379 (+ 611us) client_negotiation.cc:240] Negotiation successful
0620 00:21:43.558438 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"mutex_wait_us":81,"thread_start_us":111,"threads_started":1}
I20260620 00:21:43.559021 18340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7924463faa11468b9086a192ae23c01a" candidate_uuid: "232876eee09547499947228721aa2ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565a304ea03b4b469d1fdee368fd2334" is_pre_election: true
I20260620 00:21:43.559013 18078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7924463faa11468b9086a192ae23c01a" candidate_uuid: "232876eee09547499947228721aa2ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8d1beaf9c143349e01ea49e363c15f" is_pre_election: true
I20260620 00:21:43.559202 18340 raft_consensus.cc:2454] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 232876eee09547499947228721aa2ed7 in term 0.
I20260620 00:21:43.559231 18078 raft_consensus.cc:2454] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 232876eee09547499947228721aa2ed7 in term 0.
I20260620 00:21:43.559533 18142 leader_election.cc:302] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232876eee09547499947228721aa2ed7, 565a304ea03b4b469d1fdee368fd2334; no voters:
I20260620 00:21:43.559800 18408 raft_consensus.cc:2781] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:21:43.559887 18408 raft_consensus.cc:482] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:21:43.559927 18408 raft_consensus.cc:3037] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:43.560891 18408 raft_consensus.cc:504] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } }
I20260620 00:21:43.561077 18408 leader_election.cc:288] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [CANDIDATE]: Term 1 election: Requested vote from peers 8f8d1beaf9c143349e01ea49e363c15f (127.0.98.1:35101), 565a304ea03b4b469d1fdee368fd2334 (127.0.98.3:35735)
I20260620 00:21:43.561249 18078 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7924463faa11468b9086a192ae23c01a" candidate_uuid: "232876eee09547499947228721aa2ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f8d1beaf9c143349e01ea49e363c15f"
I20260620 00:21:43.561276 18340 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7924463faa11468b9086a192ae23c01a" candidate_uuid: "232876eee09547499947228721aa2ed7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565a304ea03b4b469d1fdee368fd2334"
I20260620 00:21:43.561342 18078 raft_consensus.cc:3037] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:43.561384 18340 raft_consensus.cc:3037] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:43.562150 18340 raft_consensus.cc:2454] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 232876eee09547499947228721aa2ed7 in term 1.
I20260620 00:21:43.562275 18078 raft_consensus.cc:2454] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 232876eee09547499947228721aa2ed7 in term 1.
I20260620 00:21:43.562335 18142 leader_election.cc:302] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 232876eee09547499947228721aa2ed7, 565a304ea03b4b469d1fdee368fd2334; no voters:
I20260620 00:21:43.562436 18408 raft_consensus.cc:2781] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:43.562654 18408 raft_consensus.cc:686] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 1 LEADER]: Becoming Leader. State: Replica: 232876eee09547499947228721aa2ed7, State: Running, Role: LEADER
I20260620 00:21:43.562769 18408 consensus_queue.cc:237] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } }
I20260620 00:21:43.563781 17936 catalog_manager.cc:5495] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 232876eee09547499947228721aa2ed7 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "232876eee09547499947228721aa2ed7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "232876eee09547499947228721aa2ed7" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33983 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 } health_report { overall_health: UNKNOWN } } }
W20260620 00:21:43.599001 18387 tablet.cc:2369] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:43.608211 18398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.605143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58076 (local address 127.0.98.2:33983)
0620 00:21:43.605224 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:43.605226 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:43.605243 (+ 17us) server_negotiation.cc:407] Connection header received
0620 00:21:43.605284 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:43.605284 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:43.605305 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:43.605378 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:21:43.605539 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.606075 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.606957 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:43.607210 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:43.607277 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:43.607661 (+ 384us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:21:43.607778 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:21:43.607855 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:43.607955 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:21:43.607977 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 00:21:43.610322 18340 raft_consensus.cc:1261] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [term 1 FOLLOWER]: Refusing update from remote peer 232876eee09547499947228721aa2ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:43.610345 18078 raft_consensus.cc:1261] T 7924463faa11468b9086a192ae23c01a P 8f8d1beaf9c143349e01ea49e363c15f [term 1 FOLLOWER]: Refusing update from remote peer 232876eee09547499947228721aa2ed7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:43.610759 18410 consensus_queue.cc:1035] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565a304ea03b4b469d1fdee368fd2334" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35735 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:43.610913 18408 consensus_queue.cc:1035] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f8d1beaf9c143349e01ea49e363c15f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:43.614231 18415 mvcc.cc:205] Tried to move back new op lower bound from 7298723445184815104 to 7298723444993794048. Current Snapshot: MvccSnapshot[applied={T|T < 7298723445184815104}]
I20260620 00:21:43.615846 17935 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:43.615973 17935 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:43.616859 17935 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 476ca4ad5cb94881be883b83d4c8a463: Sending DeleteTablet for 3 replicas of tablet 7924463faa11468b9086a192ae23c01a
I20260620 00:21:43.617218 18320 tablet_service.cc:1514] Processing DeleteTablet for tablet 7924463faa11468b9086a192ae23c01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:43 UTC) from {username='slave'} at 127.0.0.1:55466
I20260620 00:21:43.617307 18058 tablet_service.cc:1514] Processing DeleteTablet for tablet 7924463faa11468b9086a192ae23c01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:43 UTC) from {username='slave'} at 127.0.0.1:36694
I20260620 00:21:43.617465 18423 tablet_replica.cc:330] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: stopping tablet replica
I20260620 00:21:43.617540 18189 tablet_service.cc:1514] Processing DeleteTablet for tablet 7924463faa11468b9086a192ae23c01a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:43 UTC) from {username='slave'} at 127.0.0.1:58070
I20260620 00:21:43.617583 18423 raft_consensus.cc:2229] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:43.617731 18423 raft_consensus.cc:2258] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:43.617753 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 17996
I20260620 00:21:43.619732 18425 tablet_replica.cc:330] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: stopping tablet replica
I20260620 00:21:43.619849 18425 raft_consensus.cc:2229] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:21:43.619903 18423 ts_tablet_manager.cc:1905] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:43.619966 18425 raft_consensus.cc:2258] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:43.620472 18425 ts_tablet_manager.cc:1905] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:43.621855 18423 ts_tablet_manager.cc:1918] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:43.621987 18423 log.cc:1196] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/7924463faa11468b9086a192ae23c01a
I20260620 00:21:43.622242 18425 ts_tablet_manager.cc:1918] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:43.622324 18425 log.cc:1196] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/7924463faa11468b9086a192ae23c01a
I20260620 00:21:43.622381 18423 ts_tablet_manager.cc:1939] T 7924463faa11468b9086a192ae23c01a P 565a304ea03b4b469d1fdee368fd2334: Deleting consensus metadata
I20260620 00:21:43.622581 18425 ts_tablet_manager.cc:1939] T 7924463faa11468b9086a192ae23c01a P 232876eee09547499947228721aa2ed7: Deleting consensus metadata
I20260620 00:21:43.622979 17920 catalog_manager.cc:4841] TS 565a304ea03b4b469d1fdee368fd2334 (127.0.98.3:35735): tablet 7924463faa11468b9086a192ae23c01a (table test-table [id=2149375b58464597b789f39a9dc29e98]) successfully deleted
I20260620 00:21:43.622979 17923 catalog_manager.cc:4841] TS 232876eee09547499947228721aa2ed7 (127.0.98.2:33983): tablet 7924463faa11468b9086a192ae23c01a (table test-table [id=2149375b58464597b789f39a9dc29e98]) successfully deleted
W20260620 00:21:43.625922 17923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:35101 (error 108)
I20260620 00:21:43.626060 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 18127
I20260620 00:21:43.627065 18395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:43.626885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35101 (local address 127.0.0.1:36708)
0620 00:21:43.626961 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:43.627017 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 00:21:43.627269 17923 catalog_manager.cc:4584] TS 8f8d1beaf9c143349e01ea49e363c15f (127.0.98.1:35101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7924463faa11468b9086a192ae23c01a: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35101: connect: Connection refused (error 111)
I20260620 00:21:43.631273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 18258
I20260620 00:21:43.636066 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 17905
2026-06-20T00:21:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (906 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:43 dist-test-slave-r81b krb5kdc[18429](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:43 dist-test-slave-r81b krb5kdc[18429](info): set up 2 sockets
Jun 20 00:21:43 dist-test-slave-r81b krb5kdc[18429](info): commencing operation
krb5kdc: starting...
W20260620 00:21:44.728459 392 mini_kdc.cc:121] Time spent starting KDC: real 1.072s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:44 dist-test-slave-r81b krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:21:44.863149 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38427
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:44.952417 18442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:44.952649 18442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:44.952673 18442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:44.953811 18442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:44.953890 18442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:44.953907 18442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:44.953920 18442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:44.953933 18442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:44.955430 18442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:44.955746 18442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:44.955966 18442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:21:44.961172 18442 server_base.cc:1034] running on GCE node
W20260620 00:21:44.961138 18448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:44.961129 18447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:44.961120 18450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:44.961542 18442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:44.961882 18442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:44.963070 18442 hybrid_clock.cc:648] HybridClock initialized: now 1781914904963049 us; error 41 us; skew 500 ppm
Jun 20 00:21:44 dist-test-slave-r81b krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:44.965832 18442 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:21:44.966316 18442 webserver.cc:460] Webserver started at http://127.0.98.62:41443/ using document root <none> and password file <none>
I20260620 00:21:44.966559 18442 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:44.966634 18442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:44.966781 18442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:44.967898 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "cd329aabc948409493046fe1004691b7"
format_stamp: "Formatted at 2026-06-20 00:21:44 on dist-test-slave-r81b"
server_key: "6bc88c9faa8d639a911a132f0dce7f82"
server_key_iv: "8e56a5a8374e7b7ed43e947bb12f145f"
server_key_version: "encryptionkey@0"
I20260620 00:21:44.968338 18442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "cd329aabc948409493046fe1004691b7"
format_stamp: "Formatted at 2026-06-20 00:21:44 on dist-test-slave-r81b"
server_key: "6bc88c9faa8d639a911a132f0dce7f82"
server_key_iv: "8e56a5a8374e7b7ed43e947bb12f145f"
server_key_version: "encryptionkey@0"
I20260620 00:21:44.969890 18442 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:44.970785 18457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:44.970994 18442 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:44.971089 18442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "cd329aabc948409493046fe1004691b7"
format_stamp: "Formatted at 2026-06-20 00:21:44 on dist-test-slave-r81b"
server_key: "6bc88c9faa8d639a911a132f0dce7f82"
server_key_iv: "8e56a5a8374e7b7ed43e947bb12f145f"
server_key_version: "encryptionkey@0"
I20260620 00:21:44.971153 18442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:44.997553 18442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:44.999886 18442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:45.000016 18442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:45.004112 18442 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38427
I20260620 00:21:45.004191 18509 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38427 every 8 connection(s)
I20260620 00:21:45.004675 18442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:45.005697 18510 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:45.008198 18510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7: Bootstrap starting.
I20260620 00:21:45.008833 18510 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:45.008941 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 18442
I20260620 00:21:45.009104 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:45.009199 18510 log.cc:824] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:45.009352 392 external_mini_cluster.cc:1428] Setting key 41e2a6b580a749b0bb30390527e455a8
I20260620 00:21:45.010074 18510 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7: No bootstrap required, opened a new log
I20260620 00:21:45.012375 18510 raft_consensus.cc:348] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd329aabc948409493046fe1004691b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38427 } }
I20260620 00:21:45.012558 18510 raft_consensus.cc:374] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:45.012603 18510 raft_consensus.cc:729] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cd329aabc948409493046fe1004691b7, State: Initialized, Role: FOLLOWER
I20260620 00:21:45.012734 18510 consensus_queue.cc:260] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd329aabc948409493046fe1004691b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38427 } }
I20260620 00:21:45.012836 18510 raft_consensus.cc:388] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:45.012880 18510 raft_consensus.cc:482] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:45.012929 18510 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:45.013540 18510 raft_consensus.cc:504] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd329aabc948409493046fe1004691b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38427 } }
I20260620 00:21:45.013675 18510 leader_election.cc:302] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cd329aabc948409493046fe1004691b7; no voters:
I20260620 00:21:45.013893 18510 leader_election.cc:288] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:45.013949 18515 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:45.014112 18515 raft_consensus.cc:686] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [term 1 LEADER]: Becoming Leader. State: Replica: cd329aabc948409493046fe1004691b7, State: Running, Role: LEADER
I20260620 00:21:45.014178 18510 sys_catalog.cc:564] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:45.014207 18515 consensus_queue.cc:237] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd329aabc948409493046fe1004691b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38427 } }
I20260620 00:21:45.014554 18515 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cd329aabc948409493046fe1004691b7. Latest consensus state: current_term: 1 leader_uuid: "cd329aabc948409493046fe1004691b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd329aabc948409493046fe1004691b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38427 } } }
I20260620 00:21:45.014654 18515 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:45.014886 18522 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:45.014567 18516 sys_catalog.cc:455] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cd329aabc948409493046fe1004691b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cd329aabc948409493046fe1004691b7" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38427 } } }
I20260620 00:21:45.015178 18516 sys_catalog.cc:458] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:45.015407 18522 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:45.016947 18532 mvcc.cc:205] Tried to move back new op lower bound from 7298723450947301376 to 7298723450938609664. Current Snapshot: MvccSnapshot[applied={T|T < 7298723450947301376}]
I20260620 00:21:45.017423 18522 catalog_manager.cc:1347] Generated new cluster ID: f4a0854360f04cec8a6d227e21839194
I20260620 00:21:45.017475 18522 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:21:45 dist-test-slave-r81b krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:45.021910 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.009903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51288 (local address 127.0.98.62:38427)
0620 00:21:45.010067 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.010078 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.010114 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:21:45.010577 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.010584 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.010632 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.010780 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:45.011444 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.012448 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.013517 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.014201 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.014672 (+ 471us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.018030 (+ 3358us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:45.018034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:45.018035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:45.018047 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:45.020130 (+ 2083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.020569 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.020573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.020573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.020697 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.020911 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.020912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.020912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.021263 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:45.021441 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.021527 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.021681 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
I20260620 00:21:45.040349 18522 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:45.041168 18522 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:45.048494 18522 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7: Generated new TSK 0
I20260620 00:21:45.048728 18522 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:21:45.083822 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38427
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:45.171455 18536 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:45.171679 18536 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:45.171705 18536 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:45.172904 18536 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:45.172951 18536 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:45.172979 18536 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:21:45.174666 18536 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18536
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:45.174897 18536 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:45.175100 18536 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:45.180445 18541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.180460 18544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.180526 18542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:45.180804 18536 server_base.cc:1034] running on GCE node
I20260620 00:21:45.181056 18536 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:45.181349 18536 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:45.182592 18536 hybrid_clock.cc:648] HybridClock initialized: now 1781914905182577 us; error 31 us; skew 500 ppm
Jun 20 00:21:45 dist-test-slave-r81b krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:45.185691 18536 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:21:45.186218 18536 webserver.cc:460] Webserver started at http://127.0.98.1:42925/ using document root <none> and password file <none>
I20260620 00:21:45.186501 18536 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:45.186581 18536 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:45.186750 18536 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:45.187947 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "565c98cc455b4a33bc1b3531798ed64d"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "dfbce7867b1aa7790c5da3460d605b56"
server_key_iv: "d55a2486311135e7e6d859bc56087284"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.188409 18536 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "565c98cc455b4a33bc1b3531798ed64d"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "dfbce7867b1aa7790c5da3460d605b56"
server_key_iv: "d55a2486311135e7e6d859bc56087284"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.190006 18536 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:45.190840 18551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.191021 18536 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:21:45.191133 18536 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "565c98cc455b4a33bc1b3531798ed64d"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "dfbce7867b1aa7790c5da3460d605b56"
server_key_iv: "d55a2486311135e7e6d859bc56087284"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.191226 18536 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:45.201442 18536 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:45.205117 18536 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:45.205331 18536 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:45.205606 18536 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:45.205984 18536 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:45.206043 18536 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.206080 18536 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:45.206122 18536 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.212399 18536 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37273
I20260620 00:21:45.212464 18664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37273 every 8 connection(s)
I20260620 00:21:45.212790 18536 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:21:45 dist-test-slave-r81b krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:45.219511 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 18536
I20260620 00:21:45.219607 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:21:45.219810 392 external_mini_cluster.cc:1428] Setting key f596cdac51308d532677896c274a717c
I20260620 00:21:45.223380 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.213542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37859 (local address 127.0.98.62:38427)
0620 00:21:45.213589 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.213591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.213843 (+ 252us) server_negotiation.cc:407] Connection header received
0620 00:21:45.214890 (+ 1047us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.214891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.214902 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.214955 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:45.215609 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.216212 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.217257 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.217463 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.217518 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.219962 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:45.219964 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:45.219965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:45.219974 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:45.221856 (+ 1882us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.222312 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.222313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.222314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.222393 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.222649 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.222650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.222650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.222824 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:45.222909 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.223186 (+ 277us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.223283 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:21:45.223624 18667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.213518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38427 (local address 127.0.98.1:37859)
0620 00:21:45.213673 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:45.213712 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:45.214797 (+ 1085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:45.215009 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:45.215013 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:45.215279 (+ 266us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:45.215526 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.215531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.216289 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.216289 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.217157 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.217159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.217539 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.217540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.217642 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.218142 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:45.218150 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:45.219828 (+ 1678us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:45.221963 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:45.221970 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:45.221973 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:45.222207 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:45.222472 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:45.222472 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:45.222473 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:45.222574 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:45.222897 (+ 323us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:45.222903 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:45.223031 (+ 128us) client_negotiation.cc:769] Sending connection context
0620 00:21:45.223146 (+ 115us) client_negotiation.cc:240] Negotiation successful
0620 00:21:45.223339 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260620 00:21:45.224539 18665 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38427
I20260620 00:21:45.224653 18665 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:45.224848 18665 heartbeater.cc:511] Master 127.0.98.62:38427 requested a full tablet report, sending...
I20260620 00:21:45.225507 18474 ts_manager.cc:194] Registered new tserver with Master: 565c98cc455b4a33bc1b3531798ed64d (127.0.98.1:37273)
I20260620 00:21:45.226552 18474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:37859
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:21:45.253741 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38427
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:45.341516 18670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:45.341765 18670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:45.341795 18670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:45.343111 18670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:45.343173 18670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:45.343202 18670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:21:45.344898 18670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18670
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:45.345198 18670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:45.345459 18670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:45.350889 18678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.350911 18676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.350889 18675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:45.351281 18670 server_base.cc:1034] running on GCE node
I20260620 00:21:45.351701 18670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:45.352023 18670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:45.353215 18670 hybrid_clock.cc:648] HybridClock initialized: now 1781914905353209 us; error 41 us; skew 500 ppm
Jun 20 00:21:45 dist-test-slave-r81b krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:45.356083 18670 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:21:45.356628 18670 webserver.cc:460] Webserver started at http://127.0.98.2:42591/ using document root <none> and password file <none>
I20260620 00:21:45.356889 18670 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:45.356961 18670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:45.357123 18670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:45.358238 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "287bc5e16d07418092fc14119a321ed5"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "b752f0be769d6cd2774a1dbd8b28eecf"
server_key_iv: "3dff2499c74c1e2f86af5036aa5ad27f"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.358603 18670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "287bc5e16d07418092fc14119a321ed5"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "b752f0be769d6cd2774a1dbd8b28eecf"
server_key_iv: "3dff2499c74c1e2f86af5036aa5ad27f"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.360050 18670 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260620 00:21:45.360922 18685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.361186 18670 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:45.361279 18670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "287bc5e16d07418092fc14119a321ed5"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "b752f0be769d6cd2774a1dbd8b28eecf"
server_key_iv: "3dff2499c74c1e2f86af5036aa5ad27f"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.361367 18670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:45.398437 18670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:45.400998 18670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:45.401151 18670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:45.401427 18670 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:45.401767 18670 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:45.401823 18670 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.401865 18670 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:45.401892 18670 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.407805 18670 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46045
I20260620 00:21:45.407881 18798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46045 every 8 connection(s)
I20260620 00:21:45.408255 18670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:21:45.409916 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 18670
I20260620 00:21:45.410049 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:21:45.410283 392 external_mini_cluster.cc:1428] Setting key 9d78da945cb746f85d603797a102c4e5
Jun 20 00:21:45 dist-test-slave-r81b krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260620 00:21:45.419440 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.409069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51925 (local address 127.0.98.62:38427)
0620 00:21:45.409146 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.409148 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.409424 (+ 276us) server_negotiation.cc:407] Connection header received
0620 00:21:45.410570 (+ 1146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.410572 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.410585 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.410655 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:45.411454 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.412310 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.413400 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.413666 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.413737 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.416282 (+ 2545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:45.416284 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:45.416285 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:45.416294 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:45.417831 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.418317 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.418319 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.418319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.418385 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.418682 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.418683 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.418684 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.418823 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:45.418900 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.419213 (+ 313us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.419304 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260620 00:21:45.419623 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.409034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38427 (local address 127.0.98.2:51925)
0620 00:21:45.409280 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:45.409330 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:45.410419 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:45.410740 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:45.410745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:45.411130 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:45.411351 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.411357 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.412429 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.412430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.413299 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.413301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.413760 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.413761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.413915 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.414520 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:45.414529 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:45.416095 (+ 1566us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:45.417952 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:45.417961 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:45.417965 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:45.418224 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:45.418457 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:45.418458 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:45.418458 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:45.418609 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:45.418902 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:45.418909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:45.419071 (+ 162us) client_negotiation.cc:769] Sending connection context
0620 00:21:45.419179 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 00:21:45.419366 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":114,"threads_started":1}
I20260620 00:21:45.420120 18799 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38427
I20260620 00:21:45.420296 18799 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:45.420495 18799 heartbeater.cc:511] Master 127.0.98.62:38427 requested a full tablet report, sending...
I20260620 00:21:45.420894 18474 ts_manager.cc:194] Registered new tserver with Master: 287bc5e16d07418092fc14119a321ed5 (127.0.98.2:46045)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260620 00:21:45.421371 18474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:51925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:21:45.439545 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38427
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43109
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:45.524597 18804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:45.524830 18804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:45.524863 18804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:45.526185 18804 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:45.526239 18804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:45.526274 18804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:21:45.527838 18804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43109
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38427
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18804
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:45.528056 18804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:45.528338 18804 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:45.533959 18810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.533951 18809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.533970 18812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:45.534236 18804 server_base.cc:1034] running on GCE node
I20260620 00:21:45.534473 18804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:45.534814 18804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:45.536023 18804 hybrid_clock.cc:648] HybridClock initialized: now 1781914905536007 us; error 50 us; skew 500 ppm
Jun 20 00:21:45 dist-test-slave-r81b krb5kdc[18429](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:45.539011 18804 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:21:45.539575 18804 webserver.cc:460] Webserver started at http://127.0.98.3:44769/ using document root <none> and password file <none>
I20260620 00:21:45.539839 18804 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:45.539911 18804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:45.540081 18804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:45.541247 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ebbc39c6e6446e89ec1a15b770598bf"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "f97f9bd809df3c1066483564045dacc6"
server_key_iv: "744375fbcd6fa941da070eb0c1b5f9f1"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.541630 18804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ebbc39c6e6446e89ec1a15b770598bf"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "f97f9bd809df3c1066483564045dacc6"
server_key_iv: "744375fbcd6fa941da070eb0c1b5f9f1"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.543154 18804 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:21:45.544075 18819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.544317 18804 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:45.544442 18804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "3ebbc39c6e6446e89ec1a15b770598bf"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "f97f9bd809df3c1066483564045dacc6"
server_key_iv: "744375fbcd6fa941da070eb0c1b5f9f1"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.544534 18804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:45.556772 18804 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:45.559047 18804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:45.559233 18804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:45.559480 18804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:45.559803 18804 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:45.559856 18804 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.559899 18804 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:45.559926 18804 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.566480 18804 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:45455
I20260620 00:21:45.566520 18932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:45455 every 8 connection(s)
I20260620 00:21:45.566891 18804 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
Jun 20 00:21:45 dist-test-slave-r81b krb5kdc[18429](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:45.575070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 18804
I20260620 00:21:45.575176 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:21:45.575412 392 external_mini_cluster.cc:1428] Setting key d355b1f223f5163a4c621f4e2e7786ec
I20260620 00:21:45.578006 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.567764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37321 (local address 127.0.98.62:38427)
0620 00:21:45.567843 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.567845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.568020 (+ 175us) server_negotiation.cc:407] Connection header received
0620 00:21:45.569038 (+ 1018us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.569040 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.569054 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.569122 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:45.569897 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.570424 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.571475 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.571711 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.571780 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.574636 (+ 2856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:45.574637 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:45.574637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:45.574645 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:45.576603 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.577138 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.577139 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.577140 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.577201 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.577431 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.577431 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.577432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.577562 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:45.577618 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.577867 (+ 249us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.577940 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260620 00:21:45.578184 18935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.567660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38427 (local address 127.0.98.3:37321)
0620 00:21:45.567882 (+ 222us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:45.567925 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:45.568912 (+ 987us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:45.569202 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:45.569206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:45.569520 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:45.569744 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.569748 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.570519 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.570520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.571370 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.571372 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.571816 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.571817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.571963 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.572840 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:45.572883 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:45.574488 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:45.576696 (+ 2208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:45.576703 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:45.576706 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:45.577036 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:45.577272 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:45.577273 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:45.577273 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:45.577379 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:45.577622 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:45.577625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:45.577754 (+ 129us) client_negotiation.cc:769] Sending connection context
0620 00:21:45.577837 (+ 83us) client_negotiation.cc:240] Negotiation successful
0620 00:21:45.577969 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":149,"threads_started":1}
I20260620 00:21:45.578686 18933 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38427
I20260620 00:21:45.578789 18933 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:45.578943 18933 heartbeater.cc:511] Master 127.0.98.62:38427 requested a full tablet report, sending...
I20260620 00:21:45.579349 18474 ts_manager.cc:194] Registered new tserver with Master: 3ebbc39c6e6446e89ec1a15b770598bf (127.0.98.3:45455)
I20260620 00:21:45.579810 18474 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37321
I20260620 00:21:45.588014 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:21:45.596649 18514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.589606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51300 (local address 127.0.98.62:38427)
0620 00:21:45.589701 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.589704 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.589808 (+ 104us) server_negotiation.cc:407] Connection header received
0620 00:21:45.589915 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.589916 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.589926 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.589977 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:45.590556 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.591069 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.591865 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.592079 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.592205 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.593702 (+ 1497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:45.593704 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:45.593704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:45.593713 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:45.595258 (+ 1545us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.595696 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.595697 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.595698 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.595758 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:45.595984 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:45.595986 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:45.595986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:45.596190 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:45.596323 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.596468 (+ 145us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.596540 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260620 00:21:45.598582 18474 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51300:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:21:45.600327 18474 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:21:45.607565 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.603264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45455 (local address 127.0.0.1:56932)
0620 00:21:45.603707 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:45.603717 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:45.603783 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:45.604267 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:45.604268 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:45.604270 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:45.604311 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.604313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.605554 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.605555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.606482 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.606484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.607200 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.607201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.607300 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.607315 (+ 15us) client_negotiation.cc:769] Sending connection context
0620 00:21:45.607417 (+ 102us) client_negotiation.cc:240] Negotiation successful
0620 00:21:45.607479 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":99,"threads_started":1}
I20260620 00:21:45.607576 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.603488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.0.98.3:45455)
0620 00:21:45.603659 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.603664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.604002 (+ 338us) server_negotiation.cc:407] Connection header received
0620 00:21:45.604046 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.604048 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.604065 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.604157 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:45.604406 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.605456 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.606576 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.607090 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.607194 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.607270 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.607437 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.607479 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":67,"threads_started":1}
I20260620 00:21:45.607718 18945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.603229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37273 (local address 127.0.0.1:59754)
0620 00:21:45.603677 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:45.603687 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:45.603737 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:45.603918 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:45.603918 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:45.603920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:45.604125 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.604126 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.605097 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.605099 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.606480 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.606484 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.607502 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.607502 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.607561 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.607564 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:45.607608 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:21:45.607641 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":433,"mutex_wait_us":5,"thread_start_us":90,"threads_started":1}
I20260620 00:21:45.607947 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.603456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.0.98.1:37273)
0620 00:21:45.603659 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.603664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.603751 (+ 87us) server_negotiation.cc:407] Connection header received
0620 00:21:45.603812 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.603815 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.603832 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.603911 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:45.604212 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.604985 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.606589 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.607338 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.607413 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.607473 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.607816 (+ 343us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.607864 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":174,"thread_start_us":97,"threads_started":1}
I20260620 00:21:45.607746 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.603229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46045 (local address 127.0.0.1:42472)
0620 00:21:45.603446 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:45.603491 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:45.603567 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:45.603907 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:45.603910 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:45.603914 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:45.604027 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.604030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.605340 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.605341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.606495 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.606497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.607442 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.607443 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.607514 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.607518 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:45.607549 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 00:21:45.607600 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":50,"thread_start_us":75,"threads_started":1}
I20260620 00:21:45.608213 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.603460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42472 (local address 127.0.98.2:46045)
0620 00:21:45.603658 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.603663 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.603674 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:21:45.603718 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.603721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.603738 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.603954 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:45.604093 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.605231 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.606589 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.607339 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.607816 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.607886 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.607911 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.607953 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":89,"threads_started":1}
I20260620 00:21:45.608702 18867 tablet_service.cc:1467] Processing CreateTablet for tablet 56ef2fc4d4aa46789dc60364fa545c10 (DEFAULT_TABLE table=test-table [id=9c7d9b7054794b7589bd7c6ca2afa253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:45.608774 18733 tablet_service.cc:1467] Processing CreateTablet for tablet 56ef2fc4d4aa46789dc60364fa545c10 (DEFAULT_TABLE table=test-table [id=9c7d9b7054794b7589bd7c6ca2afa253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:45.608736 18599 tablet_service.cc:1467] Processing CreateTablet for tablet 56ef2fc4d4aa46789dc60364fa545c10 (DEFAULT_TABLE table=test-table [id=9c7d9b7054794b7589bd7c6ca2afa253]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:45.609010 18867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ef2fc4d4aa46789dc60364fa545c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:45.609076 18733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ef2fc4d4aa46789dc60364fa545c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:45.609076 18599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 56ef2fc4d4aa46789dc60364fa545c10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:45.611714 18950 tablet_bootstrap.cc:492] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: Bootstrap starting.
I20260620 00:21:45.612493 18950 tablet_bootstrap.cc:654] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:45.612597 18952 tablet_bootstrap.cc:492] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: Bootstrap starting.
I20260620 00:21:45.613031 18950 log.cc:824] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:45.613214 18952 tablet_bootstrap.cc:654] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:45.613794 18952 log.cc:824] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:45.613821 18950 tablet_bootstrap.cc:492] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: No bootstrap required, opened a new log
I20260620 00:21:45.613907 18950 ts_tablet_manager.cc:1397] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:21:45.614140 18951 tablet_bootstrap.cc:492] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d: Bootstrap starting.
I20260620 00:21:45.614527 18952 tablet_bootstrap.cc:492] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: No bootstrap required, opened a new log
I20260620 00:21:45.614596 18952 ts_tablet_manager.cc:1397] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:21:45.614899 18951 tablet_bootstrap.cc:654] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:45.615486 18951 log.cc:824] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:45.616012 18950 raft_consensus.cc:348] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } }
I20260620 00:21:45.616220 18950 raft_consensus.cc:374] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:45.616271 18950 raft_consensus.cc:729] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3ebbc39c6e6446e89ec1a15b770598bf, State: Initialized, Role: FOLLOWER
I20260620 00:21:45.616333 18951 tablet_bootstrap.cc:492] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d: No bootstrap required, opened a new log
I20260620 00:21:45.616406 18951 ts_tablet_manager.cc:1397] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:21:45.616390 18950 consensus_queue.cc:260] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } }
I20260620 00:21:45.616663 18950 ts_tablet_manager.cc:1428] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260620 00:21:45.616663 18933 heartbeater.cc:503] Master 127.0.98.62:38427 was elected leader, sending a full tablet report...
I20260620 00:21:45.616819 18952 raft_consensus.cc:348] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } }
I20260620 00:21:45.616984 18952 raft_consensus.cc:374] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:45.617014 18952 raft_consensus.cc:729] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 287bc5e16d07418092fc14119a321ed5, State: Initialized, Role: FOLLOWER
I20260620 00:21:45.617146 18952 consensus_queue.cc:260] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } }
I20260620 00:21:45.617496 18952 ts_tablet_manager.cc:1428] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:21:45.617632 18799 heartbeater.cc:503] Master 127.0.98.62:38427 was elected leader, sending a full tablet report...
I20260620 00:21:45.618520 18951 raft_consensus.cc:348] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } }
I20260620 00:21:45.618672 18951 raft_consensus.cc:374] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:45.618696 18951 raft_consensus.cc:729] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 565c98cc455b4a33bc1b3531798ed64d, State: Initialized, Role: FOLLOWER
I20260620 00:21:45.618808 18951 consensus_queue.cc:260] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } }
I20260620 00:21:45.619072 18665 heartbeater.cc:503] Master 127.0.98.62:38427 was elected leader, sending a full tablet report...
I20260620 00:21:45.619289 18951 ts_tablet_manager.cc:1428] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:21:45.631489 18957 raft_consensus.cc:482] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:45.631608 18957 raft_consensus.cc:504] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } }
I20260620 00:21:45.632057 18957 leader_election.cc:288] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 565c98cc455b4a33bc1b3531798ed64d (127.0.98.1:37273), 3ebbc39c6e6446e89ec1a15b770598bf (127.0.98.3:45455)
I20260620 00:21:45.635200 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.632380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:51969 (local address 127.0.98.1:37273)
0620 00:21:45.632429 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.632431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.632614 (+ 183us) server_negotiation.cc:407] Connection header received
0620 00:21:45.632647 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.632648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.632661 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.632735 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:45.632922 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.633423 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.634422 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.634820 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.634908 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.634919 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.635076 (+ 157us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.635122 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:21:45.635211 18959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.632283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37273 (local address 127.0.98.2:51969)
0620 00:21:45.632514 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:45.632525 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:45.632584 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:45.632772 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:45.632773 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:45.632775 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:45.632847 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.632849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.633503 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.633504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.634324 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.634326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.634910 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.634911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.634998 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.635001 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:45.635042 (+ 41us) client_negotiation.cc:240] Negotiation successful
0620 00:21:45.635074 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"mutex_wait_us":27,"thread_start_us":120,"threads_started":1}
I20260620 00:21:45.635247 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.632392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49401 (local address 127.0.98.3:45455)
0620 00:21:45.632437 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.632438 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.632446 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:21:45.632476 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.632477 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.632487 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.632558 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:45.632794 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.633335 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.634421 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.634823 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.634908 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.634919 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.635125 (+ 206us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.635172 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:21:45.635289 18801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.632283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:45455 (local address 127.0.98.2:49401)
0620 00:21:45.632347 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:45.632360 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:45.632405 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:45.632629 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:45.632631 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:45.632633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:45.632714 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.632717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.633412 (+ 695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.633413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.634320 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:45.634323 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.634910 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.634911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:45.635022 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.635025 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:45.635104 (+ 79us) client_negotiation.cc:240] Negotiation successful
0620 00:21:45.635145 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46,"mutex_wait_us":72}
I20260620 00:21:45.635639 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ef2fc4d4aa46789dc60364fa545c10" candidate_uuid: "287bc5e16d07418092fc14119a321ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565c98cc455b4a33bc1b3531798ed64d" is_pre_election: true
I20260620 00:21:45.635728 18887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ef2fc4d4aa46789dc60364fa545c10" candidate_uuid: "287bc5e16d07418092fc14119a321ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" is_pre_election: true
I20260620 00:21:45.635844 18887 raft_consensus.cc:2454] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287bc5e16d07418092fc14119a321ed5 in term 0.
I20260620 00:21:45.635845 18619 raft_consensus.cc:2454] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 287bc5e16d07418092fc14119a321ed5 in term 0.
I20260620 00:21:45.636312 18688 leader_election.cc:302] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287bc5e16d07418092fc14119a321ed5, 3ebbc39c6e6446e89ec1a15b770598bf; no voters:
I20260620 00:21:45.636518 18957 raft_consensus.cc:2781] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:21:45.636559 18957 raft_consensus.cc:482] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:21:45.636576 18957 raft_consensus.cc:3037] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:45.637275 18957 raft_consensus.cc:504] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } }
I20260620 00:21:45.637434 18957 leader_election.cc:288] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [CANDIDATE]: Term 1 election: Requested vote from peers 565c98cc455b4a33bc1b3531798ed64d (127.0.98.1:37273), 3ebbc39c6e6446e89ec1a15b770598bf (127.0.98.3:45455)
I20260620 00:21:45.637604 18619 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ef2fc4d4aa46789dc60364fa545c10" candidate_uuid: "287bc5e16d07418092fc14119a321ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "565c98cc455b4a33bc1b3531798ed64d"
I20260620 00:21:45.637637 18887 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "56ef2fc4d4aa46789dc60364fa545c10" candidate_uuid: "287bc5e16d07418092fc14119a321ed5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3ebbc39c6e6446e89ec1a15b770598bf"
I20260620 00:21:45.637703 18619 raft_consensus.cc:3037] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:45.637714 18887 raft_consensus.cc:3037] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:45.638635 18887 raft_consensus.cc:2454] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287bc5e16d07418092fc14119a321ed5 in term 1.
I20260620 00:21:45.638644 18619 raft_consensus.cc:2454] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 287bc5e16d07418092fc14119a321ed5 in term 1.
I20260620 00:21:45.638849 18688 leader_election.cc:302] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 287bc5e16d07418092fc14119a321ed5, 3ebbc39c6e6446e89ec1a15b770598bf; no voters:
I20260620 00:21:45.638922 18957 raft_consensus.cc:2781] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:45.639103 18957 raft_consensus.cc:686] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 1 LEADER]: Becoming Leader. State: Replica: 287bc5e16d07418092fc14119a321ed5, State: Running, Role: LEADER
I20260620 00:21:45.639200 18957 consensus_queue.cc:237] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } }
I20260620 00:21:45.640327 18474 catalog_manager.cc:5495] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 287bc5e16d07418092fc14119a321ed5 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "287bc5e16d07418092fc14119a321ed5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "287bc5e16d07418092fc14119a321ed5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46045 } health_report { overall_health: HEALTHY } } }
I20260620 00:21:45.654579 18947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.651680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42476 (local address 127.0.98.2:46045)
0620 00:21:45.651765 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.651767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.651776 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:21:45.651811 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.651812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.651833 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.651902 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:21:45.652085 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.652681 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.653533 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.653734 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.653792 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.654000 (+ 208us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:21:45.654099 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:21:45.654201 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.654446 (+ 245us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.654463 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 00:21:45.656939 18619 raft_consensus.cc:1261] T 56ef2fc4d4aa46789dc60364fa545c10 P 565c98cc455b4a33bc1b3531798ed64d [term 1 FOLLOWER]: Refusing update from remote peer 287bc5e16d07418092fc14119a321ed5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:45.656960 18887 raft_consensus.cc:1261] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [term 1 FOLLOWER]: Refusing update from remote peer 287bc5e16d07418092fc14119a321ed5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:45.657338 18960 consensus_queue.cc:1035] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "565c98cc455b4a33bc1b3531798ed64d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:45.657475 18957 consensus_queue.cc:1035] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3ebbc39c6e6446e89ec1a15b770598bf" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 45455 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:21:45.658862 18800 tablet.cc:2369] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:45.660028 18963 mvcc.cc:205] Tried to move back new op lower bound from 7298723453566763008 to 7298723453498941440. Current Snapshot: MvccSnapshot[applied={T|T < 7298723453566763008}]
I20260620 00:21:45.660686 18964 mvcc.cc:205] Tried to move back new op lower bound from 7298723453566763008 to 7298723453498941440. Current Snapshot: MvccSnapshot[applied={T|T < 7298723453566763008}]
I20260620 00:21:45.661484 18965 mvcc.cc:205] Tried to move back new op lower bound from 7298723453566763008 to 7298723453498941440. Current Snapshot: MvccSnapshot[applied={T|T < 7298723453566763008}]
I20260620 00:21:45.662395 18474 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:45.662496 18474 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51300:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:45.663210 18474 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cd329aabc948409493046fe1004691b7: Sending DeleteTablet for 3 replicas of tablet 56ef2fc4d4aa46789dc60364fa545c10
I20260620 00:21:45.663520 18599 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ef2fc4d4aa46789dc60364fa545c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59754
I20260620 00:21:45.663561 18867 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ef2fc4d4aa46789dc60364fa545c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56932
I20260620 00:21:45.663645 18733 tablet_service.cc:1514] Processing DeleteTablet for tablet 56ef2fc4d4aa46789dc60364fa545c10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:45 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42472
I20260620 00:21:45.663893 18974 tablet_replica.cc:330] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: stopping tablet replica
I20260620 00:21:45.663952 18973 tablet_replica.cc:330] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: stopping tablet replica
I20260620 00:21:45.663995 18974 raft_consensus.cc:2229] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:21:45.664029 18973 raft_consensus.cc:2229] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:45.664083 18974 raft_consensus.cc:2258] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:45.664183 18973 raft_consensus.cc:2258] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:45.664378 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 18536
I20260620 00:21:45.665755 18973 ts_tablet_manager.cc:1905] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:45.666172 18974 ts_tablet_manager.cc:1905] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:45.668087 18973 ts_tablet_manager.cc:1918] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:45.668205 18973 log.cc:1196] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/56ef2fc4d4aa46789dc60364fa545c10
I20260620 00:21:45.668130 18974 ts_tablet_manager.cc:1918] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:45.668428 18974 log.cc:1196] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/56ef2fc4d4aa46789dc60364fa545c10
I20260620 00:21:45.668500 18973 ts_tablet_manager.cc:1939] T 56ef2fc4d4aa46789dc60364fa545c10 P 3ebbc39c6e6446e89ec1a15b770598bf: Deleting consensus metadata
I20260620 00:21:45.668825 18974 ts_tablet_manager.cc:1939] T 56ef2fc4d4aa46789dc60364fa545c10 P 287bc5e16d07418092fc14119a321ed5: Deleting consensus metadata
I20260620 00:21:45.668972 18460 catalog_manager.cc:4841] TS 3ebbc39c6e6446e89ec1a15b770598bf (127.0.98.3:45455): tablet 56ef2fc4d4aa46789dc60364fa545c10 (table test-table [id=9c7d9b7054794b7589bd7c6ca2afa253]) successfully deleted
I20260620 00:21:45.669366 18458 catalog_manager.cc:4841] TS 287bc5e16d07418092fc14119a321ed5 (127.0.98.2:46045): tablet 56ef2fc4d4aa46789dc60364fa545c10 (table test-table [id=9c7d9b7054794b7589bd7c6ca2afa253]) successfully deleted
W20260620 00:21:45.671286 18461 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37273 (error 108)
I20260620 00:21:45.671479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 18670
I20260620 00:21:45.672809 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.672624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37273 (local address 127.0.0.1:59766)
0620 00:21:45.672698 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:45.672759 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37273: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260620 00:21:45.672976 18461 catalog_manager.cc:4584] TS 565c98cc455b4a33bc1b3531798ed64d (127.0.98.1:37273): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 56ef2fc4d4aa46789dc60364fa545c10: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37273: connect: Connection refused (error 111)
I20260620 00:21:45.677275 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 18804
I20260620 00:21:45.682565 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 18442
2026-06-20T00:21:45Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2047 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-06-20T00:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:45Z Disabled control of system clock
I20260620 00:21:45.711766 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44833
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:21:45.802091 18979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:45.802408 18979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:45.802433 18979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:45.803794 18979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:45.803860 18979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:45.803874 18979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:45.803887 18979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:45.803900 18979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:45.805776 18979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44833
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18979
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:45.806118 18979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:45.806509 18979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:45.812059 18985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.812120 18984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.812059 18987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:45.812453 18979 server_base.cc:1034] running on GCE node
I20260620 00:21:45.812773 18979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:45.813130 18979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:45.814312 18979 hybrid_clock.cc:648] HybridClock initialized: now 1781914905814291 us; error 31 us; skew 500 ppm
I20260620 00:21:45.815811 18979 webserver.cc:460] Webserver started at http://127.0.98.62:36779/ using document root <none> and password file <none>
I20260620 00:21:45.816085 18979 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:45.816184 18979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:45.816390 18979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:45.817503 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "d649c27807024b2d94cf730c8d0196b5"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "58c0b2d127c33a5137d726c707601270"
server_key_iv: "6e8102d4a455f79eb94bcb8e04c2f05c"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.817875 18979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "d649c27807024b2d94cf730c8d0196b5"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "58c0b2d127c33a5137d726c707601270"
server_key_iv: "6e8102d4a455f79eb94bcb8e04c2f05c"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.819399 18979 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:21:45.820425 18993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:45.820746 18979 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:45.820889 18979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "d649c27807024b2d94cf730c8d0196b5"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "58c0b2d127c33a5137d726c707601270"
server_key_iv: "6e8102d4a455f79eb94bcb8e04c2f05c"
server_key_version: "encryptionkey@0"
I20260620 00:21:45.821072 18979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:45.843648 18979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:45.844039 18979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:45.844288 18979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:45.848951 18979 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44833
I20260620 00:21:45.849045 19045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44833 every 8 connection(s)
I20260620 00:21:45.849434 18979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:45.850486 19046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:45.853022 19046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5: Bootstrap starting.
I20260620 00:21:45.853627 19046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:45.853875 19046 log.cc:824] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:45.854715 19046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5: No bootstrap required, opened a new log
I20260620 00:21:45.856632 19046 raft_consensus.cc:348] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649c27807024b2d94cf730c8d0196b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44833 } }
I20260620 00:21:45.856791 19046 raft_consensus.cc:374] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:45.856815 19046 raft_consensus.cc:729] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d649c27807024b2d94cf730c8d0196b5, State: Initialized, Role: FOLLOWER
I20260620 00:21:45.856977 19046 consensus_queue.cc:260] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649c27807024b2d94cf730c8d0196b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44833 } }
I20260620 00:21:45.857075 19046 raft_consensus.cc:388] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:45.857101 19046 raft_consensus.cc:482] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:45.857133 19046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:45.857585 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 18979
I20260620 00:21:45.857717 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:45.857770 19046 raft_consensus.cc:504] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649c27807024b2d94cf730c8d0196b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44833 } }
I20260620 00:21:45.857944 19046 leader_election.cc:302] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d649c27807024b2d94cf730c8d0196b5; no voters:
I20260620 00:21:45.857985 392 external_mini_cluster.cc:1428] Setting key 72ea98fb0de9107b1dfd0ced2d4a385a
I20260620 00:21:45.858167 19046 leader_election.cc:288] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:45.858245 19049 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:45.858420 19049 raft_consensus.cc:686] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [term 1 LEADER]: Becoming Leader. State: Replica: d649c27807024b2d94cf730c8d0196b5, State: Running, Role: LEADER
I20260620 00:21:45.858496 19046 sys_catalog.cc:564] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:45.858520 19049 consensus_queue.cc:237] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649c27807024b2d94cf730c8d0196b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44833 } }
I20260620 00:21:45.859112 19050 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d649c27807024b2d94cf730c8d0196b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649c27807024b2d94cf730c8d0196b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44833 } } }
I20260620 00:21:45.859201 19050 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:45.859411 19051 sys_catalog.cc:455] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d649c27807024b2d94cf730c8d0196b5. Latest consensus state: current_term: 1 leader_uuid: "d649c27807024b2d94cf730c8d0196b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d649c27807024b2d94cf730c8d0196b5" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44833 } } }
I20260620 00:21:45.859480 19051 sys_catalog.cc:458] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:45.859602 19060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:45.860436 19060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:45.862252 19060 catalog_manager.cc:1347] Generated new cluster ID: 063c39c58e104ddea219262af07b50e6
I20260620 00:21:45.862317 19060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:45.865103 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:45.858752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35466 (local address 127.0.98.62:44833)
0620 00:21:45.859002 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:45.859014 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:45.859335 (+ 321us) server_negotiation.cc:407] Connection header received
0620 00:21:45.859484 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:45.859492 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:45.859573 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:45.859751 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:45.860183 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.861588 (+ 1405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.862801 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:45.863419 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:45.863572 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:45.864346 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:45.864351 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:45.864351 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:45.864571 (+ 220us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:45.864617 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:45.864632 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:45.864717 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:45.864788 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 00:21:45.864878 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":71,"threads_started":1}
I20260620 00:21:45.879304 19060 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:45.879951 19060 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:45.893229 19060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5: Generated new TSK 0
I20260620 00:21:45.893476 19060 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:45.902168 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44833
--builtin_ntp_servers=127.0.98.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:21:45.989297 19070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:45.989578 19070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:45.989617 19070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:45.990906 19070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:45.990976 19070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:45.991019 19070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:21:45.992620 19070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:45.992905 19070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:45.993233 19070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:45.998667 19076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.998670 19078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:45.998705 19075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:45.998876 19070 server_base.cc:1034] running on GCE node
I20260620 00:21:45.999138 19070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:45.999362 19070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:46.000502 19070 hybrid_clock.cc:648] HybridClock initialized: now 1781914906000484 us; error 42 us; skew 500 ppm
I20260620 00:21:46.001932 19070 webserver.cc:460] Webserver started at http://127.0.98.1:44545/ using document root <none> and password file <none>
I20260620 00:21:46.002198 19070 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:46.002298 19070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:46.002453 19070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:46.003625 19070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "a1caa66b9cb8482d9868068366afcce6"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "aa829f076dc8a25c9d0a6012300e1c8d"
server_key_iv: "fdbb4370c7717bab89c8a1d243dd1a2f"
server_key_version: "encryptionkey@0"
I20260620 00:21:46.004050 19070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "a1caa66b9cb8482d9868068366afcce6"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "aa829f076dc8a25c9d0a6012300e1c8d"
server_key_iv: "fdbb4370c7717bab89c8a1d243dd1a2f"
server_key_version: "encryptionkey@0"
I20260620 00:21:46.005543 19070 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:46.006465 19084 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:46.006780 19070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:46.006901 19070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "a1caa66b9cb8482d9868068366afcce6"
format_stamp: "Formatted at 2026-06-20 00:21:45 on dist-test-slave-r81b"
server_key: "aa829f076dc8a25c9d0a6012300e1c8d"
server_key_iv: "fdbb4370c7717bab89c8a1d243dd1a2f"
server_key_version: "encryptionkey@0"
I20260620 00:21:46.006985 19070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:46.030625 19070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:46.031001 19070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:46.031172 19070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:46.031452 19070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:46.031809 19070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:46.031865 19070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:46.031901 19070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:46.031948 19070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:46.038440 19070 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41027
I20260620 00:21:46.038513 19197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41027 every 8 connection(s)
I20260620 00:21:46.039016 19070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:21:46.044999 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.039880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58157 (local address 127.0.98.62:44833)
0620 00:21:46.039967 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.039969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.040214 (+ 245us) server_negotiation.cc:407] Connection header received
0620 00:21:46.040817 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.040818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.040831 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.040892 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:46.041710 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.042291 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.043419 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.043623 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.043688 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.044674 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:46.044676 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:46.044676 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:46.044681 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:46.044697 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:46.044701 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:46.044783 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.044901 (+ 118us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.044931 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260620 00:21:46.045192 19200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.039834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44833 (local address 127.0.98.1:58157)
0620 00:21:46.040041 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:46.040078 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:46.040676 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:46.041032 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:46.041036 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:46.041044 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:46.041586 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.041590 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.042432 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.042433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.043304 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.043307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.043742 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.043743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.043874 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.044513 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:21:46.044513 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:46.044551 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:21:46.044579 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:46.044787 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:46.044792 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:46.044795 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:46.044851 (+ 56us) client_negotiation.cc:240] Negotiation successful
0620 00:21:46.044948 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":100,"threads_started":1}
I20260620 00:21:46.045818 19198 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44833
I20260620 00:21:46.045922 19198 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:46.046082 19198 heartbeater.cc:511] Master 127.0.98.62:44833 requested a full tablet report, sending...
I20260620 00:21:46.046655 19010 ts_manager.cc:194] Registered new tserver with Master: a1caa66b9cb8482d9868068366afcce6 (127.0.98.1:41027)
I20260620 00:21:46.047247 19010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:58157
I20260620 00:21:46.048714 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 19070
I20260620 00:21:46.048812 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:21:46.048974 392 external_mini_cluster.cc:1428] Setting key 80a8b52d47e28876b7204a381a2436a7
I20260620 00:21:46.049958 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44833
--builtin_ntp_servers=127.0.98.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:21:46.137475 19201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:46.137765 19201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:46.137809 19201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:46.139086 19201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:46.139159 19201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:46.139210 19201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:21:46.140730 19201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:46.141034 19201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:46.141381 19201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:46.147749 19207 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:46.147760 19209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:46.147826 19206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:46.148042 19201 server_base.cc:1034] running on GCE node
I20260620 00:21:46.148425 19201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:46.148756 19201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:46.149935 19201 hybrid_clock.cc:648] HybridClock initialized: now 1781914906149919 us; error 32 us; skew 500 ppm
I20260620 00:21:46.151789 19201 webserver.cc:460] Webserver started at http://127.0.98.2:46285/ using document root <none> and password file <none>
I20260620 00:21:46.152024 19201 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:46.152076 19201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:46.152251 19201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:46.153676 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "7217cb59300e4ceb8ab446bd344f0ef1"
format_stamp: "Formatted at 2026-06-20 00:21:46 on dist-test-slave-r81b"
server_key: "d0955fa8bb43c37b7e1bf661994a001b"
server_key_iv: "dcff1986e97e73ed205c6205a6be3c39"
server_key_version: "encryptionkey@0"
I20260620 00:21:46.154106 19201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "7217cb59300e4ceb8ab446bd344f0ef1"
format_stamp: "Formatted at 2026-06-20 00:21:46 on dist-test-slave-r81b"
server_key: "d0955fa8bb43c37b7e1bf661994a001b"
server_key_iv: "dcff1986e97e73ed205c6205a6be3c39"
server_key_version: "encryptionkey@0"
I20260620 00:21:46.155751 19201 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:21:46.156872 19215 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:46.157271 19201 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:46.157413 19201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "7217cb59300e4ceb8ab446bd344f0ef1"
format_stamp: "Formatted at 2026-06-20 00:21:46 on dist-test-slave-r81b"
server_key: "d0955fa8bb43c37b7e1bf661994a001b"
server_key_iv: "dcff1986e97e73ed205c6205a6be3c39"
server_key_version: "encryptionkey@0"
I20260620 00:21:46.157505 19201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:46.176584 19201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:46.176967 19201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:46.177140 19201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:46.177445 19201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:46.177863 19201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:46.177925 19201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:46.177965 19201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:46.177996 19201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:46.185088 19201 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46847
I20260620 00:21:46.185135 19328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46847 every 8 connection(s)
I20260620 00:21:46.185547 19201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:21:46.191598 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.186443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43817 (local address 127.0.98.62:44833)
0620 00:21:46.186582 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.186635 (+ 53us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.186796 (+ 161us) server_negotiation.cc:407] Connection header received
0620 00:21:46.187462 (+ 666us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.187464 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.187478 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.187534 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:46.188362 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.188899 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.189992 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.190208 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.190287 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.191265 (+ 978us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:46.191267 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:46.191267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:46.191274 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:46.191296 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:46.191301 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:46.191369 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.191471 (+ 102us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.191507 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260620 00:21:46.191872 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.186384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44833 (local address 127.0.98.2:43817)
0620 00:21:46.186647 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:46.186697 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:46.187314 (+ 617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:46.187636 (+ 322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:46.187640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:46.187650 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:46.188261 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.188268 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.189014 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.189016 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.189887 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.189890 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.190314 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.190315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.190470 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.191087 (+ 617us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:21:46.191088 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:46.191121 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:21:46.191148 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:46.191371 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:46.191377 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:46.191381 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:46.191475 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 00:21:46.191592 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":97,"threads_started":1}
I20260620 00:21:46.192454 19329 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44833
I20260620 00:21:46.192551 19329 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:46.192799 19329 heartbeater.cc:511] Master 127.0.98.62:44833 requested a full tablet report, sending...
I20260620 00:21:46.193238 19010 ts_manager.cc:194] Registered new tserver with Master: 7217cb59300e4ceb8ab446bd344f0ef1 (127.0.98.2:46847)
I20260620 00:21:46.193823 19010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:43817
I20260620 00:21:46.195592 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 19201
I20260620 00:21:46.195691 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:21:46.195915 392 external_mini_cluster.cc:1428] Setting key fabf75829169e9515431dc4bb3602a31
I20260620 00:21:46.197139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44833
--builtin_ntp_servers=127.0.98.20:37445
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:21:46.289752 19332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:46.289983 19332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:46.290009 19332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:46.291224 19332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:46.291266 19332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:46.291293 19332 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:21:46.292747 19332 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37445
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44833
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19332
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:46.292965 19332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:46.293251 19332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:46.298877 19338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:46.298861 19337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:46.298848 19340 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:46.299104 19332 server_base.cc:1034] running on GCE node
I20260620 00:21:46.299389 19332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:46.299681 19332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:46.300894 19332 hybrid_clock.cc:648] HybridClock initialized: now 1781914906300863 us; error 37 us; skew 500 ppm
I20260620 00:21:46.302599 19332 webserver.cc:460] Webserver started at http://127.0.98.3:45099/ using document root <none> and password file <none>
I20260620 00:21:46.302897 19332 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:46.302977 19332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:46.303122 19332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:46.304420 19332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "3eb833d31fbf4986b3317b69e9089a32"
format_stamp: "Formatted at 2026-06-20 00:21:46 on dist-test-slave-r81b"
server_key: "6a3c87b0e4d4d29a8a6cd1a893b72ffe"
server_key_iv: "77d8b230092d196cc312f0825589e0d3"
server_key_version: "encryptionkey@0"
I20260620 00:21:46.304827 19332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3eb833d31fbf4986b3317b69e9089a32"
format_stamp: "Formatted at 2026-06-20 00:21:46 on dist-test-slave-r81b"
server_key: "6a3c87b0e4d4d29a8a6cd1a893b72ffe"
server_key_iv: "77d8b230092d196cc312f0825589e0d3"
server_key_version: "encryptionkey@0"
I20260620 00:21:46.306277 19332 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:21:46.307121 19346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:46.307317 19332 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:46.307425 19332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "3eb833d31fbf4986b3317b69e9089a32"
format_stamp: "Formatted at 2026-06-20 00:21:46 on dist-test-slave-r81b"
server_key: "6a3c87b0e4d4d29a8a6cd1a893b72ffe"
server_key_iv: "77d8b230092d196cc312f0825589e0d3"
server_key_version: "encryptionkey@0"
I20260620 00:21:46.307516 19332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:46.315843 19332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:46.316267 19332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:46.316431 19332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:46.316686 19332 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:46.317045 19332 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:46.317102 19332 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:46.317239 19332 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:46.317312 19332 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:46.323892 19332 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44851
I20260620 00:21:46.323953 19459 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44851 every 8 connection(s)
I20260620 00:21:46.324362 19332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:21:46.330979 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.325263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34407 (local address 127.0.98.62:44833)
0620 00:21:46.325365 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.325368 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.325804 (+ 436us) server_negotiation.cc:407] Connection header received
0620 00:21:46.326344 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.326346 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.326357 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.326435 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:46.327323 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.327833 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.329138 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.329385 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.329469 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.330587 (+ 1118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:46.330589 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:46.330589 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:46.330595 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:46.330612 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:46.330617 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:46.330727 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.330876 (+ 149us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.330909 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:21:46.331296 19462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.325312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44833 (local address 127.0.98.3:34407)
0620 00:21:46.325618 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:46.325672 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:46.326231 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:46.326576 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:46.326580 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:46.326589 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:46.327156 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.327162 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.327974 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.327976 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.329022 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.329025 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.329528 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.329529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.329673 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.330395 (+ 722us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:21:46.330396 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:46.330433 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:21:46.330459 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:46.330758 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:46.330764 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:46.330768 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:46.330846 (+ 78us) client_negotiation.cc:240] Negotiation successful
0620 00:21:46.331016 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":137,"threads_started":1}
I20260620 00:21:46.331926 19460 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44833
I20260620 00:21:46.332026 19460 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:46.332255 19460 heartbeater.cc:511] Master 127.0.98.62:44833 requested a full tablet report, sending...
I20260620 00:21:46.332675 19010 ts_manager.cc:194] Registered new tserver with Master: 3eb833d31fbf4986b3317b69e9089a32 (127.0.98.3:44851)
I20260620 00:21:46.332768 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 19332
I20260620 00:21:46.332890 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:21:46.333148 392 external_mini_cluster.cc:1428] Setting key 4016ad9acefef8b0a046fb82b99d05d4
I20260620 00:21:46.333258 19010 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34407
I20260620 00:21:46.334268 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:21:46.339071 19053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.335803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35470 (local address 127.0.98.62:44833)
0620 00:21:46.335887 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.335890 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.336002 (+ 112us) server_negotiation.cc:407] Connection header received
0620 00:21:46.336070 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.336071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.336086 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.336152 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:46.336357 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.336857 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.337687 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.337894 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.337965 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.338757 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:46.338759 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:46.338759 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:46.338764 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:46.338781 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:46.338790 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:46.338891 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.338963 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.338975 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260620 00:21:46.340736 19010 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35470:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:21:46.342252 19010 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:21:46.350137 19472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.345032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46847 (local address 127.0.0.1:55768)
0620 00:21:46.346154 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:46.346165 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:46.346215 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:46.346428 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:46.346429 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:46.346431 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:46.346467 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.346469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.347474 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.347475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.349004 (+ 1529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.349008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.349779 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.349780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.349873 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.349890 (+ 17us) client_negotiation.cc:769] Sending connection context
0620 00:21:46.349970 (+ 80us) client_negotiation.cc:240] Negotiation successful
0620 00:21:46.350027 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1104,"mutex_wait_us":35,"thread_start_us":59,"threads_started":1}
I20260620 00:21:46.350517 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.345135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.0.98.2:46847)
0620 00:21:46.345500 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.345506 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.346280 (+ 774us) server_negotiation.cc:407] Connection header received
0620 00:21:46.346318 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.346320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.346332 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.346417 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:46.346525 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.347372 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.349133 (+ 1761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.349671 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.349802 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.349917 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.350375 (+ 458us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.350427 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":106,"threads_started":1}
I20260620 00:21:46.351167 19468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.344837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41027 (local address 127.0.0.1:52054)
0620 00:21:46.345314 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:46.345328 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:46.345443 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:46.345866 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:46.345868 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:46.345871 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:46.345939 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.345940 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.347307 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.347308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.349004 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.349008 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.350941 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.350942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.351004 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.351008 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:46.351052 (+ 44us) client_negotiation.cc:240] Negotiation successful
0620 00:21:46.351094 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"mutex_wait_us":81,"thread_start_us":43,"threads_started":1}
I20260620 00:21:46.351192 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.344974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52054 (local address 127.0.98.1:41027)
0620 00:21:46.345684 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.345689 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.345701 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:21:46.345747 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.345749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.345766 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.345848 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:46.346026 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.347209 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.349961 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.350842 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.350937 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.351042 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.351068 (+ 26us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.351111 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":645,"thread_start_us":111,"threads_started":1}
I20260620 00:21:46.351303 19263 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4dc89f102a483d8ddf516f168b4d5b (DEFAULT_TABLE table=test-table [id=44e666b8b1be462481c33743f31f2de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:46.351455 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.344759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44851 (local address 127.0.0.1:36312)
0620 00:21:46.345208 (+ 449us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:46.345258 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:46.345420 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:46.346054 (+ 634us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:46.346055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:46.346058 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:46.346119 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.346121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.347514 (+ 1393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.347515 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.350013 (+ 2498us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.350015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.351195 (+ 1180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.351196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.351268 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.351272 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:46.351338 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 00:21:46.351380 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":413,"mutex_wait_us":1507,"thread_start_us":98,"threads_started":1}
I20260620 00:21:46.351506 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.344930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36312 (local address 127.0.98.3:44851)
0620 00:21:46.345374 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.345903 (+ 529us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.345913 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:21:46.345950 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.345952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.345964 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.346055 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:46.346209 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.347335 (+ 1126us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.350125 (+ 2790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.350986 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.351060 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.351156 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.351376 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.351418 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":127,"threads_started":1}
I20260620 00:21:46.351778 19263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4dc89f102a483d8ddf516f168b4d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:46.352283 19394 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4dc89f102a483d8ddf516f168b4d5b (DEFAULT_TABLE table=test-table [id=44e666b8b1be462481c33743f31f2de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:46.352411 19132 tablet_service.cc:1467] Processing CreateTablet for tablet 1c4dc89f102a483d8ddf516f168b4d5b (DEFAULT_TABLE table=test-table [id=44e666b8b1be462481c33743f31f2de2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:46.352638 19394 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4dc89f102a483d8ddf516f168b4d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:46.352762 19132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c4dc89f102a483d8ddf516f168b4d5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:46.355327 19474 tablet_bootstrap.cc:492] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6: Bootstrap starting.
I20260620 00:21:46.355862 19475 tablet_bootstrap.cc:492] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: Bootstrap starting.
I20260620 00:21:46.356249 19476 tablet_bootstrap.cc:492] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: Bootstrap starting.
I20260620 00:21:46.356102 19474 tablet_bootstrap.cc:654] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:46.356621 19475 tablet_bootstrap.cc:654] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:46.356793 19474 log.cc:824] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:46.357014 19476 tablet_bootstrap.cc:654] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:46.357012 19475 log.cc:824] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:46.357375 19476 log.cc:824] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:46.357609 19474 tablet_bootstrap.cc:492] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6: No bootstrap required, opened a new log
I20260620 00:21:46.357692 19474 ts_tablet_manager.cc:1397] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:21:46.357846 19475 tablet_bootstrap.cc:492] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: No bootstrap required, opened a new log
I20260620 00:21:46.357913 19475 ts_tablet_manager.cc:1397] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260620 00:21:46.358054 19476 tablet_bootstrap.cc:492] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: No bootstrap required, opened a new log
I20260620 00:21:46.358134 19476 ts_tablet_manager.cc:1397] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:21:46.360896 19475 raft_consensus.cc:348] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } }
I20260620 00:21:46.361106 19475 raft_consensus.cc:374] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:46.360991 19476 raft_consensus.cc:348] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } }
I20260620 00:21:46.361151 19475 raft_consensus.cc:729] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7217cb59300e4ceb8ab446bd344f0ef1, State: Initialized, Role: FOLLOWER
I20260620 00:21:46.361196 19476 raft_consensus.cc:374] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:46.361235 19476 raft_consensus.cc:729] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3eb833d31fbf4986b3317b69e9089a32, State: Initialized, Role: FOLLOWER
I20260620 00:21:46.361186 19474 raft_consensus.cc:348] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } }
I20260620 00:21:46.361322 19474 raft_consensus.cc:374] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:46.361372 19474 raft_consensus.cc:729] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1caa66b9cb8482d9868068366afcce6, State: Initialized, Role: FOLLOWER
I20260620 00:21:46.361322 19475 consensus_queue.cc:260] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } }
I20260620 00:21:46.361403 19476 consensus_queue.cc:260] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } }
I20260620 00:21:46.361486 19474 consensus_queue.cc:260] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } }
I20260620 00:21:46.361649 19475 ts_tablet_manager.cc:1428] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260620 00:21:46.361701 19476 ts_tablet_manager.cc:1428] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260620 00:21:46.361732 19474 ts_tablet_manager.cc:1428] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260620 00:21:46.361789 19329 heartbeater.cc:503] Master 127.0.98.62:44833 was elected leader, sending a full tablet report...
I20260620 00:21:46.361900 19198 heartbeater.cc:503] Master 127.0.98.62:44833 was elected leader, sending a full tablet report...
I20260620 00:21:46.362083 19460 heartbeater.cc:503] Master 127.0.98.62:44833 was elected leader, sending a full tablet report...
W20260620 00:21:46.436185 19330 tablet.cc:2369] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:21:46.539875 19199 tablet.cc:2369] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:21:46.574990 19461 tablet.cc:2369] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:46.589299 19480 raft_consensus.cc:482] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:46.589444 19480 raft_consensus.cc:504] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } }
I20260620 00:21:46.589911 19480 leader_election.cc:288] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a1caa66b9cb8482d9868068366afcce6 (127.0.98.1:41027), 3eb833d31fbf4986b3317b69e9089a32 (127.0.98.3:44851)
I20260620 00:21:46.593196 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.590174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41027 (local address 127.0.98.2:49671)
0620 00:21:46.590271 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:46.590285 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:46.590334 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:46.590560 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:46.590561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:46.590563 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:46.590643 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.590645 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.591408 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.591409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.592347 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.592351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.592954 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.592955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.593032 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.593036 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:46.593097 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:21:46.593133 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":51}
I20260620 00:21:46.593262 19471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.590263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49671 (local address 127.0.98.1:41027)
0620 00:21:46.590321 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.590323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.590337 (+ 14us) server_negotiation.cc:407] Connection header received
0620 00:21:46.590444 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.590445 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.590458 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.590525 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:46.590797 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.591318 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.592447 (+ 1129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.592873 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.592933 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.592942 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.593111 (+ 169us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.593172 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260620 00:21:46.593408 19470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.590254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54671 (local address 127.0.98.3:44851)
0620 00:21:46.590328 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.590330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.590549 (+ 219us) server_negotiation.cc:407] Connection header received
0620 00:21:46.590585 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.590586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.590595 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.590659 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:46.590856 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.591331 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.592569 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.592968 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.593039 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.593049 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.593269 (+ 220us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.593319 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:21:46.593557 19483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.590174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44851 (local address 127.0.98.2:54671)
0620 00:21:46.590462 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:46.590473 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:46.590535 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:46.590670 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:46.590671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:46.590672 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:46.590767 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.590769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.591450 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.591451 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.592474 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:46.592477 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.593087 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.593088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:46.593178 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.593182 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:46.593242 (+ 60us) client_negotiation.cc:240] Negotiation successful
0620 00:21:46.593432 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":90,"thread_start_us":116,"threads_started":1}
I20260620 00:21:46.593623 19152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4dc89f102a483d8ddf516f168b4d5b" candidate_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1caa66b9cb8482d9868068366afcce6" is_pre_election: true
I20260620 00:21:46.593815 19152 raft_consensus.cc:2454] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7217cb59300e4ceb8ab446bd344f0ef1 in term 0.
I20260620 00:21:46.593904 19414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4dc89f102a483d8ddf516f168b4d5b" candidate_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb833d31fbf4986b3317b69e9089a32" is_pre_election: true
I20260620 00:21:46.594053 19414 raft_consensus.cc:2454] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7217cb59300e4ceb8ab446bd344f0ef1 in term 0.
I20260620 00:21:46.594122 19217 leader_election.cc:302] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7217cb59300e4ceb8ab446bd344f0ef1, a1caa66b9cb8482d9868068366afcce6; no voters:
I20260620 00:21:46.594264 19480 raft_consensus.cc:2781] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:21:46.594326 19480 raft_consensus.cc:482] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:21:46.594343 19480 raft_consensus.cc:3037] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:46.595093 19480 raft_consensus.cc:504] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } }
I20260620 00:21:46.595254 19480 leader_election.cc:288] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [CANDIDATE]: Term 1 election: Requested vote from peers a1caa66b9cb8482d9868068366afcce6 (127.0.98.1:41027), 3eb833d31fbf4986b3317b69e9089a32 (127.0.98.3:44851)
I20260620 00:21:46.595398 19152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4dc89f102a483d8ddf516f168b4d5b" candidate_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a1caa66b9cb8482d9868068366afcce6"
I20260620 00:21:46.595501 19152 raft_consensus.cc:3037] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:46.595536 19414 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c4dc89f102a483d8ddf516f168b4d5b" candidate_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3eb833d31fbf4986b3317b69e9089a32"
I20260620 00:21:46.595615 19414 raft_consensus.cc:3037] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:46.596509 19414 raft_consensus.cc:2454] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7217cb59300e4ceb8ab446bd344f0ef1 in term 1.
I20260620 00:21:46.596509 19152 raft_consensus.cc:2454] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7217cb59300e4ceb8ab446bd344f0ef1 in term 1.
I20260620 00:21:46.596705 19217 leader_election.cc:302] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7217cb59300e4ceb8ab446bd344f0ef1, a1caa66b9cb8482d9868068366afcce6; no voters:
I20260620 00:21:46.596820 19480 raft_consensus.cc:2781] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:46.596972 19480 raft_consensus.cc:686] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 1 LEADER]: Becoming Leader. State: Replica: 7217cb59300e4ceb8ab446bd344f0ef1, State: Running, Role: LEADER
I20260620 00:21:46.597108 19480 consensus_queue.cc:237] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } }
I20260620 00:21:46.598076 19008 catalog_manager.cc:5495] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7217cb59300e4ceb8ab446bd344f0ef1 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7217cb59300e4ceb8ab446bd344f0ef1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46847 } health_report { overall_health: HEALTHY } } }
I20260620 00:21:46.634979 19473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.632200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55778 (local address 127.0.98.2:46847)
0620 00:21:46.632267 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:46.632269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:46.632277 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:21:46.632313 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:46.632314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:46.632334 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:46.632403 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:21:46.632653 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.633152 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.634019 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:46.634322 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:46.634391 (+ 69us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:46.634566 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:21:46.634665 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:21:46.634767 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:46.634886 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 00:21:46.634897 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:21:46.637257 19414 raft_consensus.cc:1261] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [term 1 FOLLOWER]: Refusing update from remote peer 7217cb59300e4ceb8ab446bd344f0ef1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:46.637257 19152 raft_consensus.cc:1261] T 1c4dc89f102a483d8ddf516f168b4d5b P a1caa66b9cb8482d9868068366afcce6 [term 1 FOLLOWER]: Refusing update from remote peer 7217cb59300e4ceb8ab446bd344f0ef1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:46.637712 19484 consensus_queue.cc:1035] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a1caa66b9cb8482d9868068366afcce6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:46.637861 19480 consensus_queue.cc:1035] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3eb833d31fbf4986b3317b69e9089a32" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:46.640604 19488 mvcc.cc:205] Tried to move back new op lower bound from 7298723457582555136 to 7298723457422471168. Current Snapshot: MvccSnapshot[applied={T|T < 7298723457582555136}]
I20260620 00:21:46.641577 19489 mvcc.cc:205] Tried to move back new op lower bound from 7298723457582555136 to 7298723457422471168. Current Snapshot: MvccSnapshot[applied={T|T < 7298723457582555136}]
I20260620 00:21:46.642913 19010 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:46.643038 19010 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35470:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:46.643980 19010 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d649c27807024b2d94cf730c8d0196b5: Sending DeleteTablet for 3 replicas of tablet 1c4dc89f102a483d8ddf516f168b4d5b
I20260620 00:21:46.644369 19132 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4dc89f102a483d8ddf516f168b4d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:46 UTC) from {username='slave'} at 127.0.0.1:52054
I20260620 00:21:46.644383 19394 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4dc89f102a483d8ddf516f168b4d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:46 UTC) from {username='slave'} at 127.0.0.1:36312
I20260620 00:21:46.644622 19263 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c4dc89f102a483d8ddf516f168b4d5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:46 UTC) from {username='slave'} at 127.0.0.1:55768
I20260620 00:21:46.644675 19497 tablet_replica.cc:330] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: stopping tablet replica
I20260620 00:21:46.644781 19497 raft_consensus.cc:2229] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:46.644902 19497 raft_consensus.cc:2258] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:46.644908 19499 tablet_replica.cc:330] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: stopping tablet replica
I20260620 00:21:46.645089 19499 raft_consensus.cc:2229] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:21:46.645165 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 19070
I20260620 00:21:46.645284 19499 raft_consensus.cc:2258] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:46.645714 19497 ts_tablet_manager.cc:1905] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:46.647598 19497 ts_tablet_manager.cc:1918] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:46.647692 19497 log.cc:1196] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/1c4dc89f102a483d8ddf516f168b4d5b
I20260620 00:21:46.647984 19497 ts_tablet_manager.cc:1939] T 1c4dc89f102a483d8ddf516f168b4d5b P 3eb833d31fbf4986b3317b69e9089a32: Deleting consensus metadata
I20260620 00:21:46.648720 18996 catalog_manager.cc:4841] TS 3eb833d31fbf4986b3317b69e9089a32 (127.0.98.3:44851): tablet 1c4dc89f102a483d8ddf516f168b4d5b (table test-table [id=44e666b8b1be462481c33743f31f2de2]) successfully deleted
I20260620 00:21:46.648806 19499 ts_tablet_manager.cc:1905] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:46.650492 19499 ts_tablet_manager.cc:1918] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:46.650601 19499 log.cc:1196] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/1c4dc89f102a483d8ddf516f168b4d5b
I20260620 00:21:46.650869 19499 ts_tablet_manager.cc:1939] T 1c4dc89f102a483d8ddf516f168b4d5b P 7217cb59300e4ceb8ab446bd344f0ef1: Deleting consensus metadata
I20260620 00:21:46.651360 18995 catalog_manager.cc:4841] TS 7217cb59300e4ceb8ab446bd344f0ef1 (127.0.98.2:46847): tablet 1c4dc89f102a483d8ddf516f168b4d5b (table test-table [id=44e666b8b1be462481c33743f31f2de2]) successfully deleted
W20260620 00:21:46.655362 18995 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41027 (error 108)
I20260620 00:21:46.655745 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 19201
I20260620 00:21:46.656441 19469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:46.655860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41027 (local address 127.0.0.1:52064)
0620 00:21:46.656347 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:46.656397 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463}
W20260620 00:21:46.657486 18995 catalog_manager.cc:4584] TS a1caa66b9cb8482d9868068366afcce6 (127.0.98.1:41027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c4dc89f102a483d8ddf516f168b4d5b: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41027: connect: Connection refused (error 111)
I20260620 00:21:46.661322 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 19332
I20260620 00:21:46.666092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 18979
2026-06-20T00:21:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (982 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:46 dist-test-slave-r81b krb5kdc[19503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:46 dist-test-slave-r81b krb5kdc[19503](info): set up 2 sockets
Jun 20 00:21:46 dist-test-slave-r81b krb5kdc[19503](info): commencing operation
krb5kdc: starting...
W20260620 00:21:47.740296 392 mini_kdc.cc:121] Time spent starting KDC: real 1.054s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:47 dist-test-slave-r81b krb5kdc[19503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:21:47.875000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35003
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35003
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:47.961419 19516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:47.961648 19516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:47.961673 19516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:47.962805 19516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:47.962848 19516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:47.962863 19516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:47.962875 19516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:47.962888 19516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:47.964422 19516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35003
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35003
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:47.964638 19516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:47.964813 19516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:47.969986 19521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:47.970005 19524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:47.970036 19522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:47.970543 19516 server_base.cc:1034] running on GCE node
I20260620 00:21:47.970837 19516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:47.971155 19516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:47.972321 19516 hybrid_clock.cc:648] HybridClock initialized: now 1781914907972299 us; error 41 us; skew 500 ppm
Jun 20 00:21:47 dist-test-slave-r81b krb5kdc[19503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:47.975195 19516 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:21:47.975642 19516 webserver.cc:460] Webserver started at http://127.0.98.62:45069/ using document root <none> and password file <none>
I20260620 00:21:47.975843 19516 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:47.975885 19516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:47.975978 19516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:47.977104 19516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "2e859a99b6884d5ea101556e847a4b58"
format_stamp: "Formatted at 2026-06-20 00:21:47 on dist-test-slave-r81b"
server_key: "9dc7f2c1c89bbf877fe69be8cb90ac63"
server_key_iv: "bf140af8224507df1846549d156a98bb"
server_key_version: "encryptionkey@0"
I20260620 00:21:47.977458 19516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e859a99b6884d5ea101556e847a4b58"
format_stamp: "Formatted at 2026-06-20 00:21:47 on dist-test-slave-r81b"
server_key: "9dc7f2c1c89bbf877fe69be8cb90ac63"
server_key_iv: "bf140af8224507df1846549d156a98bb"
server_key_version: "encryptionkey@0"
I20260620 00:21:47.978854 19516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:21:47.979596 19531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:47.979900 19516 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:47.980011 19516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "2e859a99b6884d5ea101556e847a4b58"
format_stamp: "Formatted at 2026-06-20 00:21:47 on dist-test-slave-r81b"
server_key: "9dc7f2c1c89bbf877fe69be8cb90ac63"
server_key_iv: "bf140af8224507df1846549d156a98bb"
server_key_version: "encryptionkey@0"
I20260620 00:21:47.980096 19516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:47.993538 19516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:47.995844 19516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:47.996047 19516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:48.000339 19516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35003
I20260620 00:21:48.000399 19583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35003 every 8 connection(s)
I20260620 00:21:48.000928 19516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:48.002010 19584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:48.004633 19584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58: Bootstrap starting.
I20260620 00:21:48.005313 19584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:48.005640 19584 log.cc:824] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:48.006512 19584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58: No bootstrap required, opened a new log
I20260620 00:21:48.008373 19584 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e859a99b6884d5ea101556e847a4b58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } }
I20260620 00:21:48.008536 19584 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:48.008565 19584 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e859a99b6884d5ea101556e847a4b58, State: Initialized, Role: FOLLOWER
I20260620 00:21:48.008721 19584 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e859a99b6884d5ea101556e847a4b58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } }
I20260620 00:21:48.008805 19584 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:48.008838 19584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:48.008904 19584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:48.009585 19584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e859a99b6884d5ea101556e847a4b58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } }
I20260620 00:21:48.009684 19584 leader_election.cc:302] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e859a99b6884d5ea101556e847a4b58; no voters:
I20260620 00:21:48.009902 19584 leader_election.cc:288] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:48.010005 19587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:48.010221 19584 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:48.010236 19587 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [term 1 LEADER]: Becoming Leader. State: Replica: 2e859a99b6884d5ea101556e847a4b58, State: Running, Role: LEADER
I20260620 00:21:48.010353 19587 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e859a99b6884d5ea101556e847a4b58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } }
I20260620 00:21:48.010809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 19516
I20260620 00:21:48.010962 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:48.010952 19589 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e859a99b6884d5ea101556e847a4b58. Latest consensus state: current_term: 1 leader_uuid: "2e859a99b6884d5ea101556e847a4b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e859a99b6884d5ea101556e847a4b58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } } }
I20260620 00:21:48.011037 19589 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:48.011191 392 external_mini_cluster.cc:1428] Setting key b7edd8ebe2b195ad55ccb1c2e1ba8649
I20260620 00:21:48.011720 19596 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:48.011790 19588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e859a99b6884d5ea101556e847a4b58" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e859a99b6884d5ea101556e847a4b58" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35003 } } }
I20260620 00:21:48.011886 19588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:48.012290 19596 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:48.014078 19596 catalog_manager.cc:1347] Generated new cluster ID: e77573e7963a4b34b78663f6934ab25a
I20260620 00:21:48.014138 19596 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:21:48 dist-test-slave-r81b krb5kdc[19503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914907, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:48.022696 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.011714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.0.98.62:35003)
0620 00:21:48.012280 (+ 566us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.012291 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.012328 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:21:48.012475 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.012485 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.012558 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.012766 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:48.013561 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.014728 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.015544 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.016017 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.016149 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.018612 (+ 2463us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:48.018617 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:48.018617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:48.018632 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:48.020968 (+ 2336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.021384 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.021387 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.021388 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.021497 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.021758 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.021759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.021759 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.022037 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:48.022224 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.022301 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.022457 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":73,"threads_started":1}
I20260620 00:21:48.048625 19596 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:48.049304 19596 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:48.052367 19596 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58: Generated new TSK 0
I20260620 00:21:48.052826 19596 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260620 00:21:48.084829 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:48.192036 19610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:48.192298 19610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:48.192327 19610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:48.193588 19610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:48.193637 19610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:48.193666 19610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:21:48.195225 19610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:48.195484 19610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:48.195729 19610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:48.201586 19616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:48.201606 19618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:48.201560 19615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:48.201876 19610 server_base.cc:1034] running on GCE node
I20260620 00:21:48.202152 19610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:48.202489 19610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:48.203706 19610 hybrid_clock.cc:648] HybridClock initialized: now 1781914908203691 us; error 31 us; skew 500 ppm
Jun 20 00:21:48 dist-test-slave-r81b krb5kdc[19503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:48.206602 19610 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260620 00:21:48.207185 19610 webserver.cc:460] Webserver started at http://127.0.98.1:41297/ using document root <none> and password file <none>
I20260620 00:21:48.207460 19610 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:48.207535 19610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:48.207690 19610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:48.208910 19610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "b3724957f4084b76adde2ce07ea14eb8"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "3ab6d811021fe3f4c1ae61c6e2456a39"
server_key_iv: "93f1dee792ce73a634c2fddd516d3a7f"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.209345 19610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b3724957f4084b76adde2ce07ea14eb8"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "3ab6d811021fe3f4c1ae61c6e2456a39"
server_key_iv: "93f1dee792ce73a634c2fddd516d3a7f"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.210912 19610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:21:48.211818 19625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.212004 19610 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:48.212103 19610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "b3724957f4084b76adde2ce07ea14eb8"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "3ab6d811021fe3f4c1ae61c6e2456a39"
server_key_iv: "93f1dee792ce73a634c2fddd516d3a7f"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.212242 19610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:48.222534 19610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:48.224843 19610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:48.225023 19610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:48.225276 19610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:48.225610 19610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:48.225667 19610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.225714 19610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:48.225770 19610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.232247 19610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37163
I20260620 00:21:48.232321 19738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37163 every 8 connection(s)
I20260620 00:21:48.232662 19610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
Jun 20 00:21:48 dist-test-slave-r81b krb5kdc[19503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:48.241771 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 19610
I20260620 00:21:48.241901 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:21:48.242156 392 external_mini_cluster.cc:1428] Setting key 109cf23b2835c9deeb844becc86f4013
I20260620 00:21:48.242969 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.233479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58335 (local address 127.0.98.62:35003)
0620 00:21:48.233551 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.233553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.233850 (+ 297us) server_negotiation.cc:407] Connection header received
0620 00:21:48.234777 (+ 927us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.234779 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.234794 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.234854 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:48.235585 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.236103 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.237192 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.237410 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.237478 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.239807 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:48.239809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:48.239809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:48.239821 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:48.241167 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.241601 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.241602 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.241603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.241674 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.242047 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.242049 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.242049 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.242206 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:48.242282 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.242738 (+ 456us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.242864 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:21:48.243089 19741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.233461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35003 (local address 127.0.98.1:58335)
0620 00:21:48.233694 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:48.233734 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:48.234663 (+ 929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:48.234936 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:48.234940 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:48.235213 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:48.235442 (+ 229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.235446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.236257 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.236258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.237086 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.237088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.237494 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.237495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.237610 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.238168 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:48.238177 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:48.239714 (+ 1537us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:48.241270 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:48.241277 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:48.241279 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:48.241519 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:48.241806 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:48.241807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:48.241808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:48.241977 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:48.242386 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:48.242393 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:48.242590 (+ 197us) client_negotiation.cc:769] Sending connection context
0620 00:21:48.242688 (+ 98us) client_negotiation.cc:240] Negotiation successful
0620 00:21:48.242866 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":143,"threads_started":1}
I20260620 00:21:48.243643 19739 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35003
I20260620 00:21:48.243775 19739 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:48.243969 19739 heartbeater.cc:511] Master 127.0.98.62:35003 requested a full tablet report, sending...
I20260620 00:21:48.244580 19548 ts_manager.cc:194] Registered new tserver with Master: b3724957f4084b76adde2ce07ea14eb8 (127.0.98.1:37163)
I20260620 00:21:48.245244 19548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58335
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260620 00:21:48.271082 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:48.356503 19744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:48.356791 19744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:48.356837 19744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:48.358147 19744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:48.358196 19744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:48.358261 19744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:21:48.359902 19744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:48.360249 19744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:48.360512 19744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:48.365986 19752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:48.365996 19750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:48.366001 19749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:48.366459 19744 server_base.cc:1034] running on GCE node
I20260620 00:21:48.366703 19744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:48.366987 19744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:48.368244 19744 hybrid_clock.cc:648] HybridClock initialized: now 1781914908368227 us; error 49 us; skew 500 ppm
Jun 20 00:21:48 dist-test-slave-r81b krb5kdc[19503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:48.371322 19744 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260620 00:21:48.371903 19744 webserver.cc:460] Webserver started at http://127.0.98.2:35997/ using document root <none> and password file <none>
I20260620 00:21:48.372228 19744 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:48.372309 19744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:48.372479 19744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:48.373627 19744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "255bbf631e474e70bfe005898969ceee"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "1f6b3285582d5f084ce4d770dac6d5e4"
server_key_iv: "ac27e56037044a7ca21fe74b23c5af93"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.374094 19744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "255bbf631e474e70bfe005898969ceee"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "1f6b3285582d5f084ce4d770dac6d5e4"
server_key_iv: "ac27e56037044a7ca21fe74b23c5af93"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.375816 19744 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260620 00:21:48.376821 19759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.377020 19744 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260620 00:21:48.377128 19744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "255bbf631e474e70bfe005898969ceee"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "1f6b3285582d5f084ce4d770dac6d5e4"
server_key_iv: "ac27e56037044a7ca21fe74b23c5af93"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.377223 19744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:48.388499 19744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:48.390875 19744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:48.391050 19744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:48.391320 19744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:48.391649 19744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:48.391709 19744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.391805 19744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:48.391832 19744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.398718 19744 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39933
I20260620 00:21:48.398756 19872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39933 every 8 connection(s)
I20260620 00:21:48.399111 19744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
Jun 20 00:21:48 dist-test-slave-r81b krb5kdc[19503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:48.406692 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 19744
I20260620 00:21:48.406831 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:21:48.407013 392 external_mini_cluster.cc:1428] Setting key 354118af7207752266cefd5af0ecffce
I20260620 00:21:48.410288 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.399951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43961 (local address 127.0.98.62:35003)
0620 00:21:48.400022 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.400025 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.400197 (+ 172us) server_negotiation.cc:407] Connection header received
0620 00:21:48.401193 (+ 996us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.401194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.401205 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.401278 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:48.402008 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.402520 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.403630 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.403925 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.403994 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.406496 (+ 2502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:48.406498 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:48.406498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:48.406507 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:48.408604 (+ 2097us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.409069 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.409070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.409071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.409146 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.409425 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.409426 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.409426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.409582 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:48.409697 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.410095 (+ 398us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.410213 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:21:48.410571 19875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.399879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35003 (local address 127.0.98.2:43961)
0620 00:21:48.400047 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:48.400092 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:48.401075 (+ 983us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:48.401391 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:48.401395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:48.401687 (+ 292us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:48.401889 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.401893 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.402625 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.402627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.403518 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.403521 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.404013 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.404014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.404131 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.404863 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:48.404872 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:48.406391 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:48.408715 (+ 2324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:48.408724 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:48.408726 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:48.408985 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:48.409220 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:48.409221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:48.409221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:48.409356 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:48.409696 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:48.409705 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:48.409918 (+ 213us) client_negotiation.cc:769] Sending connection context
0620 00:21:48.410059 (+ 141us) client_negotiation.cc:240] Negotiation successful
0620 00:21:48.410290 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260620 00:21:48.411206 19873 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35003
I20260620 00:21:48.411324 19873 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:48.411514 19873 heartbeater.cc:511] Master 127.0.98.62:35003 requested a full tablet report, sending...
I20260620 00:21:48.411950 19548 ts_manager.cc:194] Registered new tserver with Master: 255bbf631e474e70bfe005898969ceee (127.0.98.2:39933)
I20260620 00:21:48.412533 19548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43961
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260620 00:21:48.435988 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46291
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:48.521097 19878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:48.521380 19878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:48.521425 19878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:48.522682 19878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:48.522763 19878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:48.522819 19878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:21:48.524497 19878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46291
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35003
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:48.524830 19878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:48.525085 19878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:48.530573 19886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:48.530568 19883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:48.530680 19878 server_base.cc:1034] running on GCE node
W20260620 00:21:48.530568 19884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:48.531083 19878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:48.531355 19878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:48.532531 19878 hybrid_clock.cc:648] HybridClock initialized: now 1781914908532507 us; error 40 us; skew 500 ppm
Jun 20 00:21:48 dist-test-slave-r81b krb5kdc[19503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:48.535471 19878 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260620 00:21:48.535948 19878 webserver.cc:460] Webserver started at http://127.0.98.3:36677/ using document root <none> and password file <none>
I20260620 00:21:48.536216 19878 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:48.536268 19878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:48.536370 19878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:48.537474 19878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "8d05bd60d964497d8d2a82ca33429260"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "5d81845dd0cd707bdb4e2337c57c3a93"
server_key_iv: "122e1e464a266ecb9df74d16b670c604"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.537825 19878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "8d05bd60d964497d8d2a82ca33429260"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "5d81845dd0cd707bdb4e2337c57c3a93"
server_key_iv: "122e1e464a266ecb9df74d16b670c604"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.539177 19878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:21:48.540210 19893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.540405 19878 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:48.540514 19878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "8d05bd60d964497d8d2a82ca33429260"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "5d81845dd0cd707bdb4e2337c57c3a93"
server_key_iv: "122e1e464a266ecb9df74d16b670c604"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.540599 19878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:48.554528 19878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:48.557184 19878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:48.557365 19878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:48.557638 19878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:48.557973 19878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:48.558028 19878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.558080 19878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:48.558111 19878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.564566 19878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33375
I20260620 00:21:48.564620 20006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33375 every 8 connection(s)
I20260620 00:21:48.564998 19878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:21:48.571448 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 19878
I20260620 00:21:48.571595 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:21:48.571820 392 external_mini_cluster.cc:1428] Setting key 77abae77fae75a51f164091def5610b9
Jun 20 00:21:48 dist-test-slave-r81b krb5kdc[19503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:21:48.576606 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.565814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47551 (local address 127.0.98.62:35003)
0620 00:21:48.565899 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.565902 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.566802 (+ 900us) server_negotiation.cc:407] Connection header received
0620 00:21:48.567896 (+ 1094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.567898 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.567912 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.567981 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:48.568795 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.569370 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.570526 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.570754 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.570852 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.573653 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:48.573655 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:48.573655 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:48.573662 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:48.575211 (+ 1549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.575632 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.575633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.575633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.575686 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.575936 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.575937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.575937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.576045 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:48.576095 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.576464 (+ 369us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.576535 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 00:21:48.576894 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.565811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35003 (local address 127.0.98.3:47551)
0620 00:21:48.566657 (+ 846us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:48.566703 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:48.567757 (+ 1054us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:48.568063 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:48.568067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:48.568462 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:48.568685 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.568691 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.569485 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.569487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.570420 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.570422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.570865 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.570866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.571017 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.571611 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:21:48.571623 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:48.573549 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:48.575304 (+ 1755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:48.575311 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:48.575313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:48.575560 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:48.575771 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:21:48.575772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:21:48.575772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:21:48.575880 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:21:48.576097 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:48.576101 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:48.576333 (+ 232us) client_negotiation.cc:769] Sending connection context
0620 00:21:48.576447 (+ 114us) client_negotiation.cc:240] Negotiation successful
0620 00:21:48.576607 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":782,"thread_start_us":78,"threads_started":1}
I20260620 00:21:48.577441 20007 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35003
I20260620 00:21:48.577554 20007 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:48.577775 20007 heartbeater.cc:511] Master 127.0.98.62:35003 requested a full tablet report, sending...
I20260620 00:21:48.578179 19548 ts_manager.cc:194] Registered new tserver with Master: 8d05bd60d964497d8d2a82ca33429260 (127.0.98.3:33375)
I20260620 00:21:48.578599 19548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:47551
I20260620 00:21:48.584606 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:21:48.593366 19604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.586050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56760 (local address 127.0.98.62:35003)
0620 00:21:48.586147 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.586150 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.586246 (+ 96us) server_negotiation.cc:407] Connection header received
0620 00:21:48.586371 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.586372 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.586385 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.586461 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:48.587114 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.587644 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.588537 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.588738 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.588838 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.590351 (+ 1513us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:48.590353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:48.590353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:48.590362 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:48.591926 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.592421 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.592423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.592423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.592496 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:48.592729 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:48.592730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:48.592730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:48.592907 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:48.593027 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.593157 (+ 130us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.593254 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260620 00:21:48.595279 19548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:21:48.597148 19548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:21:48.604482 20021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.600066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33375 (local address 127.0.0.1:57020)
0620 00:21:48.600501 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:48.600510 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:48.600562 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:48.600908 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:48.600909 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:48.600910 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:48.600987 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.600989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.602372 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.602373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.603307 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.603310 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.604084 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.604085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.604194 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.604212 (+ 18us) client_negotiation.cc:769] Sending connection context
0620 00:21:48.604293 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 00:21:48.604359 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":420,"mutex_wait_us":135,"thread_start_us":110,"threads_started":1}
I20260620 00:21:48.604521 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.599962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52584 (local address 127.0.98.2:39933)
0620 00:21:48.600291 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.600297 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.600309 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:21:48.600471 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.600474 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.600493 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.600579 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:48.601120 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.601846 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.603387 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.604176 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.604253 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.604311 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.604380 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.604420 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":110,"threads_started":1}
I20260620 00:21:48.604628 20019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.599860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37163 (local address 127.0.0.1:47544)
0620 00:21:48.600360 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:48.600370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:48.600414 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:48.600871 (+ 457us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:48.600874 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:48.600876 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:48.600987 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.600990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.602352 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.602353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.603451 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.603453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.604387 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.604388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.604450 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.604454 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:48.604485 (+ 31us) client_negotiation.cc:240] Negotiation successful
0620 00:21:48.604528 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"mutex_wait_us":47,"thread_start_us":84,"threads_started":1}
I20260620 00:21:48.604660 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.600197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57020 (local address 127.0.98.3:33375)
0620 00:21:48.600378 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.600383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.600647 (+ 264us) server_negotiation.cc:407] Connection header received
0620 00:21:48.600701 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.600704 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.600725 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.600846 (+ 121us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:48.601078 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.602241 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.603405 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.603985 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.604082 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.604151 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.604522 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.604570 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":87,"threads_started":1}
I20260620 00:21:48.604482 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.599829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39933 (local address 127.0.0.1:52584)
0620 00:21:48.599993 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:48.600035 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:48.600102 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:48.600961 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:48.600962 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:48.600964 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:48.601023 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.601025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.601961 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.601962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.603289 (+ 1327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.603292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.604251 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.604252 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.604314 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.604317 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:48.604366 (+ 49us) client_negotiation.cc:240] Negotiation successful
0620 00:21:48.604401 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":1,"thread_start_us":86,"threads_started":1}
I20260620 00:21:48.605198 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.600229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47544 (local address 127.0.98.1:37163)
0620 00:21:48.600678 (+ 449us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.600683 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.600692 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:21:48.600732 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.600734 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.600749 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.601062 (+ 313us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:48.601143 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.602242 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.603537 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.604292 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.604998 (+ 706us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.605074 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.605105 (+ 31us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.605141 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":413,"thread_start_us":102,"threads_started":1}
I20260620 00:21:48.605734 19941 tablet_service.cc:1467] Processing CreateTablet for tablet 524eda9265324e3db01275fb59d26eba (DEFAULT_TABLE table=test-table [id=744eab4da41f429980e9766a002aac47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:48.605851 19673 tablet_service.cc:1467] Processing CreateTablet for tablet 524eda9265324e3db01275fb59d26eba (DEFAULT_TABLE table=test-table [id=744eab4da41f429980e9766a002aac47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:48.606040 19941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524eda9265324e3db01275fb59d26eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:48.606235 19673 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524eda9265324e3db01275fb59d26eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:48.609037 20024 tablet_bootstrap.cc:492] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: Bootstrap starting.
I20260620 00:21:48.608947 19807 tablet_service.cc:1467] Processing CreateTablet for tablet 524eda9265324e3db01275fb59d26eba (DEFAULT_TABLE table=test-table [id=744eab4da41f429980e9766a002aac47]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:48.609344 19807 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 524eda9265324e3db01275fb59d26eba. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:48.609387 20025 tablet_bootstrap.cc:492] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8: Bootstrap starting.
I20260620 00:21:48.609862 20024 tablet_bootstrap.cc:654] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:48.610033 20025 tablet_bootstrap.cc:654] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:48.610600 20024 log.cc:824] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:48.610706 20025 log.cc:824] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:48.611539 20024 tablet_bootstrap.cc:492] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: No bootstrap required, opened a new log
I20260620 00:21:48.611588 20025 tablet_bootstrap.cc:492] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8: No bootstrap required, opened a new log
I20260620 00:21:48.611646 20024 ts_tablet_manager.cc:1397] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:21:48.611649 20025 ts_tablet_manager.cc:1397] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:21:48.613135 20028 tablet_bootstrap.cc:492] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: Bootstrap starting.
I20260620 00:21:48.613251 20024 raft_consensus.cc:348] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } }
I20260620 00:21:48.613438 20024 raft_consensus.cc:374] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:48.613490 20024 raft_consensus.cc:729] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d05bd60d964497d8d2a82ca33429260, State: Initialized, Role: FOLLOWER
I20260620 00:21:48.613649 20024 consensus_queue.cc:260] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } }
I20260620 00:21:48.613952 20024 ts_tablet_manager.cc:1428] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260620 00:21:48.614040 20028 tablet_bootstrap.cc:654] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:48.613968 20025 raft_consensus.cc:348] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } }
I20260620 00:21:48.614145 20007 heartbeater.cc:503] Master 127.0.98.62:35003 was elected leader, sending a full tablet report...
I20260620 00:21:48.614164 20025 raft_consensus.cc:374] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:48.614203 20025 raft_consensus.cc:729] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3724957f4084b76adde2ce07ea14eb8, State: Initialized, Role: FOLLOWER
I20260620 00:21:48.614355 20025 consensus_queue.cc:260] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } }
I20260620 00:21:48.614727 19739 heartbeater.cc:503] Master 127.0.98.62:35003 was elected leader, sending a full tablet report...
I20260620 00:21:48.614814 20028 log.cc:824] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:48.614727 20025 ts_tablet_manager.cc:1428] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260620 00:21:48.615696 20028 tablet_bootstrap.cc:492] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: No bootstrap required, opened a new log
I20260620 00:21:48.615777 20028 ts_tablet_manager.cc:1397] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260620 00:21:48.617278 20028 raft_consensus.cc:348] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } }
I20260620 00:21:48.617415 20028 raft_consensus.cc:374] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:48.617475 20028 raft_consensus.cc:729] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 255bbf631e474e70bfe005898969ceee, State: Initialized, Role: FOLLOWER
I20260620 00:21:48.617605 20028 consensus_queue.cc:260] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } }
I20260620 00:21:48.617906 20028 ts_tablet_manager.cc:1428] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260620 00:21:48.617914 19873 heartbeater.cc:503] Master 127.0.98.62:35003 was elected leader, sending a full tablet report...
I20260620 00:21:48.644940 20029 raft_consensus.cc:482] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:48.645112 20029 raft_consensus.cc:504] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } }
I20260620 00:21:48.645485 20029 leader_election.cc:288] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 255bbf631e474e70bfe005898969ceee (127.0.98.2:39933), b3724957f4084b76adde2ce07ea14eb8 (127.0.98.1:37163)
I20260620 00:21:48.648396 20009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.645693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39933 (local address 127.0.98.3:51757)
0620 00:21:48.645750 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:48.645764 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:48.645807 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:48.645960 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:48.645960 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:48.645962 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:48.646057 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.646059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.646714 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.646715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.647478 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.647481 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.648077 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.648078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.648205 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.648214 (+ 9us) client_negotiation.cc:769] Sending connection context
0620 00:21:48.648277 (+ 63us) client_negotiation.cc:240] Negotiation successful
0620 00:21:48.648320 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 00:21:48.648396 20020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.645782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51757 (local address 127.0.98.2:39933)
0620 00:21:48.645841 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.645843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.645853 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:21:48.645881 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.645882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.645893 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.645946 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:48.646136 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.646609 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.647582 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.647959 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.648065 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.648077 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.648282 (+ 205us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.648321 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:21:48.648670 20023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.645900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:49369 (local address 127.0.98.1:37163)
0620 00:21:48.645984 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.645987 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.646120 (+ 133us) server_negotiation.cc:407] Connection header received
0620 00:21:48.646221 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.646222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.646241 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.646309 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:48.646480 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.646945 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.647897 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.648315 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.648397 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.648407 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.648565 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.648611 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260620 00:21:48.648679 20033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.645800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37163 (local address 127.0.98.3:49369)
0620 00:21:48.646041 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:48.646050 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:48.646131 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:48.646316 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:48.646317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:48.646319 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:48.646399 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.646402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.647035 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.647036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.647799 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:48.647801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.648415 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.648416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:48.648491 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.648496 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:21:48.648557 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:21:48.648604 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":144,"threads_started":1}
I20260620 00:21:48.648777 19827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524eda9265324e3db01275fb59d26eba" candidate_uuid: "8d05bd60d964497d8d2a82ca33429260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "255bbf631e474e70bfe005898969ceee" is_pre_election: true
I20260620 00:21:48.648937 19827 raft_consensus.cc:2454] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d05bd60d964497d8d2a82ca33429260 in term 0.
I20260620 00:21:48.648972 19693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524eda9265324e3db01275fb59d26eba" candidate_uuid: "8d05bd60d964497d8d2a82ca33429260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3724957f4084b76adde2ce07ea14eb8" is_pre_election: true
I20260620 00:21:48.649140 19693 raft_consensus.cc:2454] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d05bd60d964497d8d2a82ca33429260 in term 0.
I20260620 00:21:48.649185 19896 leader_election.cc:302] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 255bbf631e474e70bfe005898969ceee, 8d05bd60d964497d8d2a82ca33429260; no voters:
I20260620 00:21:48.649340 20029 raft_consensus.cc:2781] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:21:48.649407 20029 raft_consensus.cc:482] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:21:48.649444 20029 raft_consensus.cc:3037] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 0 FOLLOWER]: Advancing to term 1
W20260620 00:21:48.649680 19874 tablet.cc:2369] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:48.650157 20029 raft_consensus.cc:504] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } }
I20260620 00:21:48.650308 20029 leader_election.cc:288] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [CANDIDATE]: Term 1 election: Requested vote from peers 255bbf631e474e70bfe005898969ceee (127.0.98.2:39933), b3724957f4084b76adde2ce07ea14eb8 (127.0.98.1:37163)
I20260620 00:21:48.650455 19827 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524eda9265324e3db01275fb59d26eba" candidate_uuid: "8d05bd60d964497d8d2a82ca33429260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "255bbf631e474e70bfe005898969ceee"
I20260620 00:21:48.650494 19693 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "524eda9265324e3db01275fb59d26eba" candidate_uuid: "8d05bd60d964497d8d2a82ca33429260" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3724957f4084b76adde2ce07ea14eb8"
I20260620 00:21:48.650540 19827 raft_consensus.cc:3037] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:48.650563 19693 raft_consensus.cc:3037] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:48.651444 19693 raft_consensus.cc:2454] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d05bd60d964497d8d2a82ca33429260 in term 1.
I20260620 00:21:48.651444 19827 raft_consensus.cc:2454] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d05bd60d964497d8d2a82ca33429260 in term 1.
I20260620 00:21:48.651611 19896 leader_election.cc:302] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 255bbf631e474e70bfe005898969ceee, 8d05bd60d964497d8d2a82ca33429260; no voters:
I20260620 00:21:48.651698 20029 raft_consensus.cc:2781] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:48.651852 20029 raft_consensus.cc:686] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 1 LEADER]: Becoming Leader. State: Replica: 8d05bd60d964497d8d2a82ca33429260, State: Running, Role: LEADER
I20260620 00:21:48.651957 20029 consensus_queue.cc:237] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } }
I20260620 00:21:48.652987 19548 catalog_manager.cc:5495] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d05bd60d964497d8d2a82ca33429260 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "8d05bd60d964497d8d2a82ca33429260" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d05bd60d964497d8d2a82ca33429260" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33375 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 } health_report { overall_health: UNKNOWN } } }
I20260620 00:21:48.669613 20022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.666993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57036 (local address 127.0.98.3:33375)
0620 00:21:48.667099 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.667101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.667110 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:21:48.667145 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.667146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.667167 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.667243 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:21:48.667451 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.667977 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.668816 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.669004 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.669056 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.669221 (+ 165us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:21:48.669340 (+ 119us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:21:48.669407 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.669488 (+ 81us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.669508 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260620 00:21:48.671657 19827 raft_consensus.cc:1261] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [term 1 FOLLOWER]: Refusing update from remote peer 8d05bd60d964497d8d2a82ca33429260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:48.671682 19693 raft_consensus.cc:1261] T 524eda9265324e3db01275fb59d26eba P b3724957f4084b76adde2ce07ea14eb8 [term 1 FOLLOWER]: Refusing update from remote peer 8d05bd60d964497d8d2a82ca33429260: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:48.672075 20034 consensus_queue.cc:1035] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "255bbf631e474e70bfe005898969ceee" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:48.672327 20029 consensus_queue.cc:1035] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3724957f4084b76adde2ce07ea14eb8" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:48.675159 20038 mvcc.cc:205] Tried to move back new op lower bound from 7298723465916022784 to 7298723465839243264. Current Snapshot: MvccSnapshot[applied={T|T < 7298723465916022784}]
I20260620 00:21:48.677397 19547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:48.677517 19547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56760:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:48.678344 19547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2e859a99b6884d5ea101556e847a4b58: Sending DeleteTablet for 3 replicas of tablet 524eda9265324e3db01275fb59d26eba
I20260620 00:21:48.678682 19673 tablet_service.cc:1514] Processing DeleteTablet for tablet 524eda9265324e3db01275fb59d26eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47544
I20260620 00:21:48.678752 19807 tablet_service.cc:1514] Processing DeleteTablet for tablet 524eda9265324e3db01275fb59d26eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52584
I20260620 00:21:48.679237 20048 tablet_replica.cc:330] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: stopping tablet replica
I20260620 00:21:48.679234 19941 tablet_service.cc:1514] Processing DeleteTablet for tablet 524eda9265324e3db01275fb59d26eba with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:48 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57020
I20260620 00:21:48.679369 20048 raft_consensus.cc:2229] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:48.679498 20049 tablet_replica.cc:330] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: stopping tablet replica
I20260620 00:21:48.679500 20048 raft_consensus.cc:2258] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:48.679597 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 19610
I20260620 00:21:48.680678 20048 ts_tablet_manager.cc:1905] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:48.681035 20049 raft_consensus.cc:2229] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:21:48.681222 20049 raft_consensus.cc:2258] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:48.681672 20049 ts_tablet_manager.cc:1905] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:48.682744 20048 ts_tablet_manager.cc:1918] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:48.682832 20048 log.cc:1196] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/524eda9265324e3db01275fb59d26eba
I20260620 00:21:48.683135 20048 ts_tablet_manager.cc:1939] T 524eda9265324e3db01275fb59d26eba P 255bbf631e474e70bfe005898969ceee: Deleting consensus metadata
I20260620 00:21:48.683657 19534 catalog_manager.cc:4841] TS 255bbf631e474e70bfe005898969ceee (127.0.98.2:39933): tablet 524eda9265324e3db01275fb59d26eba (table test-table [id=744eab4da41f429980e9766a002aac47]) successfully deleted
I20260620 00:21:48.684036 20049 ts_tablet_manager.cc:1918] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:48.684160 20049 log.cc:1196] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/524eda9265324e3db01275fb59d26eba
I20260620 00:21:48.684470 20049 ts_tablet_manager.cc:1939] T 524eda9265324e3db01275fb59d26eba P 8d05bd60d964497d8d2a82ca33429260: Deleting consensus metadata
I20260620 00:21:48.684917 19534 catalog_manager.cc:4841] TS 8d05bd60d964497d8d2a82ca33429260 (127.0.98.3:33375): tablet 524eda9265324e3db01275fb59d26eba (table test-table [id=744eab4da41f429980e9766a002aac47]) successfully deleted
W20260620 00:21:48.686826 19532 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37163 (error 108)
I20260620 00:21:48.687011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 19744
I20260620 00:21:48.688176 20018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.687990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37163 (local address 127.0.0.1:47558)
0620 00:21:48.688065 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:48.688114 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37163: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260620 00:21:48.689073 19532 catalog_manager.cc:4584] TS b3724957f4084b76adde2ce07ea14eb8 (127.0.98.1:37163): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 524eda9265324e3db01275fb59d26eba: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37163: connect: Connection refused (error 111)
I20260620 00:21:48.692819 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 19878
I20260620 00:21:48.698011 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 19516
2026-06-20T00:21:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2032 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-06-20T00:21:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:48Z Disabled control of system clock
I20260620 00:21:48.726397 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45021
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45021
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:21:48.811775 20054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:48.812032 20054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:48.812059 20054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:48.813467 20054 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:48.813552 20054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:48.813567 20054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:48.813581 20054 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:48.813594 20054 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:48.815090 20054 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45021
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:48.815330 20054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:48.815599 20054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:48.821023 20060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:48.821154 20054 server_base.cc:1034] running on GCE node
W20260620 00:21:48.821044 20062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:48.821044 20059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:48.821548 20054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:48.821894 20054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:48.823105 20054 hybrid_clock.cc:648] HybridClock initialized: now 1781914908823056 us; error 85 us; skew 500 ppm
I20260620 00:21:48.824668 20054 webserver.cc:460] Webserver started at http://127.0.98.62:38517/ using document root <none> and password file <none>
I20260620 00:21:48.824972 20054 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:48.825054 20054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:48.825212 20054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:48.826360 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "0901b3bfd55f4897b07d8238306059fa"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "b8bc4115824562a041cb31234ce2260f"
server_key_iv: "826fa722a3eec3a2db89d16395792c15"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.826745 20054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "0901b3bfd55f4897b07d8238306059fa"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "b8bc4115824562a041cb31234ce2260f"
server_key_iv: "826fa722a3eec3a2db89d16395792c15"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.828336 20054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260620 00:21:48.829217 20068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.829452 20054 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:48.829603 20054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "0901b3bfd55f4897b07d8238306059fa"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "b8bc4115824562a041cb31234ce2260f"
server_key_iv: "826fa722a3eec3a2db89d16395792c15"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.829691 20054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:48.847831 20054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:48.848258 20054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:48.848423 20054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:48.852313 20054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45021
I20260620 00:21:48.852381 20120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45021 every 8 connection(s)
I20260620 00:21:48.852763 20054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:48.853807 20121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:48.856166 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa: Bootstrap starting.
I20260620 00:21:48.856705 20121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:48.856961 20121 log.cc:824] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:48.857654 20121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa: No bootstrap required, opened a new log
I20260620 00:21:48.859231 20121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901b3bfd55f4897b07d8238306059fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } }
I20260620 00:21:48.859373 20121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:48.859396 20121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0901b3bfd55f4897b07d8238306059fa, State: Initialized, Role: FOLLOWER
I20260620 00:21:48.859546 20121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901b3bfd55f4897b07d8238306059fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } }
I20260620 00:21:48.859627 20121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:48.859649 20121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:48.859678 20121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:48.860230 20121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901b3bfd55f4897b07d8238306059fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } }
I20260620 00:21:48.860320 20121 leader_election.cc:302] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0901b3bfd55f4897b07d8238306059fa; no voters:
I20260620 00:21:48.860447 20121 leader_election.cc:288] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:48.860543 20124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:48.860719 20124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [term 1 LEADER]: Becoming Leader. State: Replica: 0901b3bfd55f4897b07d8238306059fa, State: Running, Role: LEADER
I20260620 00:21:48.860766 20121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:48.860837 20124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901b3bfd55f4897b07d8238306059fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } }
I20260620 00:21:48.861393 20126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0901b3bfd55f4897b07d8238306059fa. Latest consensus state: current_term: 1 leader_uuid: "0901b3bfd55f4897b07d8238306059fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901b3bfd55f4897b07d8238306059fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } } }
I20260620 00:21:48.861510 20126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:48.861677 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 20054
I20260620 00:21:48.861786 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:48.861928 20125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0901b3bfd55f4897b07d8238306059fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0901b3bfd55f4897b07d8238306059fa" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45021 } } }
I20260620 00:21:48.861991 392 external_mini_cluster.cc:1428] Setting key 92966b3fa86f488a6be11b0966c80c25
I20260620 00:21:48.862027 20125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:48.862182 20133 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:48.862910 20133 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:48.864600 20133 catalog_manager.cc:1347] Generated new cluster ID: 76ff4498a67747bc982ea479ef56cf78
I20260620 00:21:48.864657 20133 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:48.869953 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:48.862478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37692 (local address 127.0.98.62:45021)
0620 00:21:48.862717 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:48.862726 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:48.862758 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:21:48.862879 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:48.862885 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:48.862963 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:48.863122 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:48.864388 (+ 1266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.865845 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.867140 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:48.867827 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:48.867961 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:48.869007 (+ 1046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:48.869012 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:48.869013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:48.869250 (+ 237us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:48.869298 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:48.869315 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:48.869380 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:48.869624 (+ 244us) server_negotiation.cc:299] Negotiation successful
0620 00:21:48.869718 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":99,"threads_started":1}
I20260620 00:21:48.872694 20133 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:48.873412 20133 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:48.877885 20133 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa: Generated new TSK 0
I20260620 00:21:48.878099 20133 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:48.880641 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--builtin_ntp_servers=127.0.98.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260620 00:21:48.970245 20145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:48.970470 20145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:48.970495 20145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:48.971758 20145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:48.971805 20145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:48.971834 20145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260620 00:21:48.973336 20145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:48.973598 20145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:48.973917 20145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:48.979194 20150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:48.979194 20153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:48.979215 20151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:48.979490 20145 server_base.cc:1034] running on GCE node
I20260620 00:21:48.979732 20145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:48.980059 20145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:48.981225 20145 hybrid_clock.cc:648] HybridClock initialized: now 1781914908981207 us; error 51 us; skew 500 ppm
I20260620 00:21:48.982720 20145 webserver.cc:460] Webserver started at http://127.0.98.1:40339/ using document root <none> and password file <none>
I20260620 00:21:48.982990 20145 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:48.983062 20145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:48.983237 20145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:48.984488 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/data/instance:
uuid: "72c4f432cd184494b0d49d786b22302b"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "c1dec19ab2062fbe6a8894846818614d"
server_key_iv: "913f612afef8cf9317d24d3a6ff54a2f"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.984871 20145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/wal/instance:
uuid: "72c4f432cd184494b0d49d786b22302b"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "c1dec19ab2062fbe6a8894846818614d"
server_key_iv: "913f612afef8cf9317d24d3a6ff54a2f"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.986284 20145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:21:48.987083 20159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:48.987247 20145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:21:48.987346 20145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/wal
uuid: "72c4f432cd184494b0d49d786b22302b"
format_stamp: "Formatted at 2026-06-20 00:21:48 on dist-test-slave-r81b"
server_key: "c1dec19ab2062fbe6a8894846818614d"
server_key_iv: "913f612afef8cf9317d24d3a6ff54a2f"
server_key_version: "encryptionkey@0"
I20260620 00:21:48.987425 20145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:49.005712 20145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:49.006086 20145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:49.006260 20145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:49.006517 20145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:49.006982 20145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:49.007018 20145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:49.007041 20145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:49.007088 20145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:49.013414 20145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43525
I20260620 00:21:49.013450 20272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43525 every 8 connection(s)
I20260620 00:21:49.013906 20145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/data/info.pb
I20260620 00:21:49.016052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 20145
I20260620 00:21:49.016237 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-0/wal/instance
I20260620 00:21:49.016427 392 external_mini_cluster.cc:1428] Setting key ebf4ebb0982c059440a2beae42324b67
I20260620 00:21:49.017563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--builtin_ntp_servers=127.0.98.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:21:49.020476 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.014776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55477 (local address 127.0.98.62:45021)
0620 00:21:49.014827 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.014829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.015115 (+ 286us) server_negotiation.cc:407] Connection header received
0620 00:21:49.015699 (+ 584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.015701 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.015716 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.015777 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:49.016788 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.017551 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.018920 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.019143 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.019224 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.020180 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:49.020182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:49.020182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:49.020188 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:49.020204 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:49.020211 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:49.020280 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.020357 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.020393 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:21:49.020748 20275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.014779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45021 (local address 127.0.98.1:55477)
0620 00:21:49.014976 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:49.015016 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:49.015575 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:49.015860 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:49.015864 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:49.015894 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:49.016592 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.016614 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.017699 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.017700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.018824 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.018827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.019235 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.019236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.019396 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.019985 (+ 589us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:21:49.019985 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:49.020018 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:21:49.020043 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:49.020280 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:49.020288 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:49.020291 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:49.020363 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:21:49.020474 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260620 00:21:49.021322 20273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45021
I20260620 00:21:49.021456 20273 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:49.021703 20273 heartbeater.cc:511] Master 127.0.98.62:45021 requested a full tablet report, sending...
I20260620 00:21:49.022370 20085 ts_manager.cc:194] Registered new tserver with Master: 72c4f432cd184494b0d49d786b22302b (127.0.98.1:43525)
I20260620 00:21:49.023439 20085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:55477
W20260620 00:21:49.105010 20276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:49.105257 20276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:49.105281 20276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:49.106480 20276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:49.106523 20276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:49.106549 20276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260620 00:21:49.107913 20276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:49.108196 20276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:49.108515 20276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:49.114409 20282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:49.114456 20284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:49.114409 20281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:49.114728 20276 server_base.cc:1034] running on GCE node
I20260620 00:21:49.114969 20276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:49.115279 20276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:49.116457 20276 hybrid_clock.cc:648] HybridClock initialized: now 1781914909116436 us; error 38 us; skew 500 ppm
I20260620 00:21:49.117863 20276 webserver.cc:460] Webserver started at http://127.0.98.2:42827/ using document root <none> and password file <none>
I20260620 00:21:49.118124 20276 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:49.118196 20276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:49.118352 20276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:49.119484 20276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/data/instance:
uuid: "bc465edf63ad4e69a0b8b9d02378f282"
format_stamp: "Formatted at 2026-06-20 00:21:49 on dist-test-slave-r81b"
server_key: "87b55a37a26f951276d5770b2a26f529"
server_key_iv: "1bf34c0a80263a3fa16246b45b43b7cb"
server_key_version: "encryptionkey@0"
I20260620 00:21:49.119870 20276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/wal/instance:
uuid: "bc465edf63ad4e69a0b8b9d02378f282"
format_stamp: "Formatted at 2026-06-20 00:21:49 on dist-test-slave-r81b"
server_key: "87b55a37a26f951276d5770b2a26f529"
server_key_iv: "1bf34c0a80263a3fa16246b45b43b7cb"
server_key_version: "encryptionkey@0"
I20260620 00:21:49.121349 20276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:21:49.122267 20290 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:49.122566 20276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:49.122669 20276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/wal
uuid: "bc465edf63ad4e69a0b8b9d02378f282"
format_stamp: "Formatted at 2026-06-20 00:21:49 on dist-test-slave-r81b"
server_key: "87b55a37a26f951276d5770b2a26f529"
server_key_iv: "1bf34c0a80263a3fa16246b45b43b7cb"
server_key_version: "encryptionkey@0"
I20260620 00:21:49.122753 20276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:49.131364 20276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:49.131640 20276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:49.131747 20276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:49.131930 20276 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:49.132754 20276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:49.132799 20276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:49.132823 20276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:49.132836 20276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:49.139030 20276 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37903
I20260620 00:21:49.139133 20403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37903 every 8 connection(s)
I20260620 00:21:49.139456 20276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/data/info.pb
I20260620 00:21:49.142949 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 20276
I20260620 00:21:49.143072 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/wal/instance
I20260620 00:21:49.143263 392 external_mini_cluster.cc:1428] Setting key ad9f701d8845bf385cff5d21000cdf03
I20260620 00:21:49.144411 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--builtin_ntp_servers=127.0.98.20:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260620 00:21:49.145591 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.140256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54425 (local address 127.0.98.62:45021)
0620 00:21:49.140329 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.140331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.140614 (+ 283us) server_negotiation.cc:407] Connection header received
0620 00:21:49.141110 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.141111 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.141122 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.141179 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:49.141842 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.142358 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.143852 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.144086 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.144216 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.145263 (+ 1047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:49.145267 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:49.145267 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:49.145278 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:49.145292 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:49.145296 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:49.145362 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.145466 (+ 104us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.145505 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260620 00:21:49.145876 20406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.140264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45021 (local address 127.0.98.2:54425)
0620 00:21:49.140473 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:49.140509 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:49.140998 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:49.141233 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:49.141236 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:49.141244 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:49.141754 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.141758 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.142467 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.142468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.143754 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.143757 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.144229 (+ 472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.144230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.144385 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.145068 (+ 683us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:21:49.145068 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:49.145100 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:21:49.145127 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:49.145365 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:49.145372 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:49.145376 (+ 4us) client_negotiation.cc:769] Sending connection context
0620 00:21:49.145470 (+ 94us) client_negotiation.cc:240] Negotiation successful
0620 00:21:49.145596 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":89,"threads_started":1}
I20260620 00:21:49.146509 20404 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45021
I20260620 00:21:49.146629 20404 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:49.146880 20404 heartbeater.cc:511] Master 127.0.98.62:45021 requested a full tablet report, sending...
I20260620 00:21:49.147308 20085 ts_manager.cc:194] Registered new tserver with Master: bc465edf63ad4e69a0b8b9d02378f282 (127.0.98.2:37903)
I20260620 00:21:49.147698 20085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:54425
W20260620 00:21:49.227368 20407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:49.227660 20407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:49.227687 20407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:49.229107 20407 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:49.229159 20407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:49.229189 20407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260620 00:21:49.230728 20407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45021
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20407
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:49.231029 20407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:49.231356 20407 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:49.236903 20413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:49.236982 20412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:49.236903 20415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:49.237429 20407 server_base.cc:1034] running on GCE node
I20260620 00:21:49.237689 20407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:49.237977 20407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:49.239166 20407 hybrid_clock.cc:648] HybridClock initialized: now 1781914909239150 us; error 25 us; skew 500 ppm
I20260620 00:21:49.241236 20407 webserver.cc:460] Webserver started at http://127.0.98.3:40311/ using document root <none> and password file <none>
I20260620 00:21:49.241541 20407 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:49.241595 20407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:49.241780 20407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:49.242928 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/data/instance:
uuid: "ac95ff0c71d2449f8c2b201cbeb499e0"
format_stamp: "Formatted at 2026-06-20 00:21:49 on dist-test-slave-r81b"
server_key: "e2e023148d1fd735168cf34aecdbfff0"
server_key_iv: "5a31a6760d14cdfb7b8f48f9b37e3690"
server_key_version: "encryptionkey@0"
I20260620 00:21:49.243315 20407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ac95ff0c71d2449f8c2b201cbeb499e0"
format_stamp: "Formatted at 2026-06-20 00:21:49 on dist-test-slave-r81b"
server_key: "e2e023148d1fd735168cf34aecdbfff0"
server_key_iv: "5a31a6760d14cdfb7b8f48f9b37e3690"
server_key_version: "encryptionkey@0"
I20260620 00:21:49.244827 20407 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:21:49.245698 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:49.246064 20407 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:49.246201 20407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/wal
uuid: "ac95ff0c71d2449f8c2b201cbeb499e0"
format_stamp: "Formatted at 2026-06-20 00:21:49 on dist-test-slave-r81b"
server_key: "e2e023148d1fd735168cf34aecdbfff0"
server_key_iv: "5a31a6760d14cdfb7b8f48f9b37e3690"
server_key_version: "encryptionkey@0"
I20260620 00:21:49.246291 20407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:49.271811 20407 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:49.272229 20407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:49.272395 20407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:49.272656 20407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260620 00:21:49.273006 20407 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260620 00:21:49.273061 20407 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:49.273102 20407 ts_tablet_manager.cc:610] Registered 0 tablets
I20260620 00:21:49.273125 20407 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:49.279048 20407 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:36501
I20260620 00:21:49.279090 20534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:36501 every 8 connection(s)
I20260620 00:21:49.279449 20407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/data/info.pb
I20260620 00:21:49.280292 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 20407
I20260620 00:21:49.280409 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/wal/instance
I20260620 00:21:49.280629 392 external_mini_cluster.cc:1428] Setting key c8ca093ea735fd1f3ca6d960c6f1d5da
I20260620 00:21:49.285036 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.280215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48269 (local address 127.0.98.62:45021)
0620 00:21:49.280298 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.280301 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.280581 (+ 280us) server_negotiation.cc:407] Connection header received
0620 00:21:49.281330 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.281331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.281343 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.281397 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:49.282157 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.282663 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.283658 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.283860 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.283923 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.284802 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:49.284803 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:49.284803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:49.284808 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:49.284823 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:49.284827 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:49.284873 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.284950 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.284975 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260620 00:21:49.285308 20537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.280228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45021 (local address 127.0.98.3:48269)
0620 00:21:49.280426 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:49.280471 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:49.281221 (+ 750us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:49.281472 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:49.281476 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:49.281486 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:21:49.282067 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.282072 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.282756 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.282756 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.283571 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.283573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.283945 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.283946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.284056 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.284667 (+ 611us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0620 00:21:49.284667 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:21:49.284697 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0620 00:21:49.284721 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:21:49.284883 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:21:49.284887 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:21:49.284890 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:49.284951 (+ 61us) client_negotiation.cc:240] Negotiation successful
0620 00:21:49.285034 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":97,"threads_started":1}
I20260620 00:21:49.285913 20535 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45021
I20260620 00:21:49.286063 20535 heartbeater.cc:464] Registering TS with master...
I20260620 00:21:49.286262 20535 heartbeater.cc:511] Master 127.0.98.62:45021 requested a full tablet report, sending...
I20260620 00:21:49.286726 20085 ts_manager.cc:194] Registered new tserver with Master: ac95ff0c71d2449f8c2b201cbeb499e0 (127.0.98.3:36501)
I20260620 00:21:49.287061 20085 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:48269
I20260620 00:21:49.292742 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260620 00:21:49.297163 20142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.294157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37700 (local address 127.0.98.62:45021)
0620 00:21:49.294227 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.294229 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.294371 (+ 142us) server_negotiation.cc:407] Connection header received
0620 00:21:49.294426 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.294426 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.294438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.294490 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:49.294647 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.295207 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.296006 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.296278 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.296334 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.296963 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:49.296964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:49.296965 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:49.296969 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:49.296983 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:49.296986 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:49.297025 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.297076 (+ 51us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.297099 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:21:49.299099 20085 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37700:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260620 00:21:49.300840 20085 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260620 00:21:49.307966 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.303503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47828 (local address 127.0.98.3:36501)
0620 00:21:49.303717 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.303723 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.304012 (+ 289us) server_negotiation.cc:407] Connection header received
0620 00:21:49.304082 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.304085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.304098 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.304255 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:49.304515 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.305352 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.306739 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.307521 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.307603 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.307709 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.307824 (+ 115us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.307866 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20260620 00:21:49.307960 20545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.303339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36501 (local address 127.0.0.1:47828)
0620 00:21:49.303884 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:49.303929 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:49.304010 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:49.304346 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:49.304350 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:49.304353 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:49.304428 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.304430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.305441 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.305442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.306643 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.306646 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.307627 (+ 981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.307628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.307712 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.307726 (+ 14us) client_negotiation.cc:769] Sending connection context
0620 00:21:49.307807 (+ 81us) client_negotiation.cc:240] Negotiation successful
0620 00:21:49.307866 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":83,"threads_started":1}
I20260620 00:21:49.308166 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.303633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36518 (local address 127.0.98.1:43525)
0620 00:21:49.303863 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.303869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.304285 (+ 416us) server_negotiation.cc:407] Connection header received
0620 00:21:49.304354 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.304356 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.304370 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.304446 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:49.304668 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.305831 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.307051 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.307591 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.307664 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.307771 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.308001 (+ 230us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.308047 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":152,"threads_started":1}
I20260620 00:21:49.308319 20544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.303370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43525 (local address 127.0.0.1:36518)
0620 00:21:49.304231 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:49.304240 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:49.304285 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:49.304526 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:49.304527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:49.304529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:49.304576 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.304578 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.305940 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.305941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.306965 (+ 1024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.306967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.307805 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.307806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.307859 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.307862 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:49.307909 (+ 47us) client_negotiation.cc:240] Negotiation successful
0620 00:21:49.307942 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":845,"mutex_wait_us":34,"thread_start_us":109,"threads_started":1}
I20260620 00:21:49.308831 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.303339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37903 (local address 127.0.0.1:36464)
0620 00:21:49.304095 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:49.304102 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:49.304429 (+ 327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:49.304652 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:49.304653 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:49.304656 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:49.304702 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.304704 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.305965 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.305966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.307131 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.307133 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.308491 (+ 1358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.308492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.308565 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.308570 (+ 5us) client_negotiation.cc:769] Sending connection context
0620 00:21:49.308629 (+ 59us) client_negotiation.cc:240] Negotiation successful
0620 00:21:49.308671 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":748,"mutex_wait_us":27,"thread_start_us":75,"threads_started":1}
I20260620 00:21:49.308883 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.303642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36464 (local address 127.0.98.2:37903)
0620 00:21:49.303879 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.303883 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.304183 (+ 300us) server_negotiation.cc:407] Connection header received
0620 00:21:49.304532 (+ 349us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.304535 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.304549 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.304647 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:49.304796 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.305877 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.307230 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.307951 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.308020 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.308453 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.308725 (+ 272us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.308775 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":131,"threads_started":1}
I20260620 00:21:49.309146 20469 tablet_service.cc:1467] Processing CreateTablet for tablet a4ae5acc08cd43508585aac7c351a224 (DEFAULT_TABLE table=test-table [id=3ac8d94198ad4a26a1e7736019dcb1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:49.309522 20469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ae5acc08cd43508585aac7c351a224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:49.309640 20338 tablet_service.cc:1467] Processing CreateTablet for tablet a4ae5acc08cd43508585aac7c351a224 (DEFAULT_TABLE table=test-table [id=3ac8d94198ad4a26a1e7736019dcb1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:49.309985 20338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ae5acc08cd43508585aac7c351a224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:49.312466 20549 tablet_bootstrap.cc:492] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: Bootstrap starting.
I20260620 00:21:49.312883 20550 tablet_bootstrap.cc:492] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: Bootstrap starting.
I20260620 00:21:49.312815 20207 tablet_service.cc:1467] Processing CreateTablet for tablet a4ae5acc08cd43508585aac7c351a224 (DEFAULT_TABLE table=test-table [id=3ac8d94198ad4a26a1e7736019dcb1e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260620 00:21:49.313105 20207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4ae5acc08cd43508585aac7c351a224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:49.313333 20549 tablet_bootstrap.cc:654] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:49.313539 20550 tablet_bootstrap.cc:654] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:49.313671 20549 log.cc:824] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:49.313856 20550 log.cc:824] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:49.314489 20549 tablet_bootstrap.cc:492] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: No bootstrap required, opened a new log
I20260620 00:21:49.314575 20549 ts_tablet_manager.cc:1397] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:21:49.314652 20550 tablet_bootstrap.cc:492] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: No bootstrap required, opened a new log
I20260620 00:21:49.314728 20550 ts_tablet_manager.cc:1397] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:21:49.316691 20553 tablet_bootstrap.cc:492] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b: Bootstrap starting.
I20260620 00:21:49.316851 20549 raft_consensus.cc:348] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } }
I20260620 00:21:49.317030 20549 raft_consensus.cc:374] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:49.317077 20549 raft_consensus.cc:729] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ac95ff0c71d2449f8c2b201cbeb499e0, State: Initialized, Role: FOLLOWER
I20260620 00:21:49.317198 20549 consensus_queue.cc:260] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } }
I20260620 00:21:49.317446 20553 tablet_bootstrap.cc:654] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:49.317472 20549 ts_tablet_manager.cc:1428] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260620 00:21:49.317407 20550 raft_consensus.cc:348] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } }
I20260620 00:21:49.317570 20550 raft_consensus.cc:374] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:49.317616 20535 heartbeater.cc:503] Master 127.0.98.62:45021 was elected leader, sending a full tablet report...
I20260620 00:21:49.317615 20550 raft_consensus.cc:729] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bc465edf63ad4e69a0b8b9d02378f282, State: Initialized, Role: FOLLOWER
I20260620 00:21:49.317731 20550 consensus_queue.cc:260] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } }
I20260620 00:21:49.317799 20553 log.cc:824] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:49.317992 20550 ts_tablet_manager.cc:1428] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260620 00:21:49.318014 20404 heartbeater.cc:503] Master 127.0.98.62:45021 was elected leader, sending a full tablet report...
I20260620 00:21:49.319026 20553 tablet_bootstrap.cc:492] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b: No bootstrap required, opened a new log
I20260620 00:21:49.319110 20553 ts_tablet_manager.cc:1397] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260620 00:21:49.321107 20553 raft_consensus.cc:348] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } }
I20260620 00:21:49.321238 20553 raft_consensus.cc:374] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:49.321260 20553 raft_consensus.cc:729] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72c4f432cd184494b0d49d786b22302b, State: Initialized, Role: FOLLOWER
I20260620 00:21:49.321374 20553 consensus_queue.cc:260] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } }
I20260620 00:21:49.321645 20553 ts_tablet_manager.cc:1428] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260620 00:21:49.321753 20273 heartbeater.cc:503] Master 127.0.98.62:45021 was elected leader, sending a full tablet report...
W20260620 00:21:49.390028 20405 tablet.cc:2369] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:21:49.514788 20274 tablet.cc:2369] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260620 00:21:49.530117 20536 tablet.cc:2369] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260620 00:21:49.597018 20557 raft_consensus.cc:482] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260620 00:21:49.597158 20557 raft_consensus.cc:504] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } }
I20260620 00:21:49.597559 20557 leader_election.cc:288] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bc465edf63ad4e69a0b8b9d02378f282 (127.0.98.2:37903), ac95ff0c71d2449f8c2b201cbeb499e0 (127.0.98.3:36501)
I20260620 00:21:49.602035 20546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.597852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40037 (local address 127.0.98.3:36501)
0620 00:21:49.597926 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.597928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.598056 (+ 128us) server_negotiation.cc:407] Connection header received
0620 00:21:49.598094 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.598095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.598106 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.598156 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:49.598351 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.599110 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.600877 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.601489 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.601583 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.601596 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.601895 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.601949 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:21:49.602044 20547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.597889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35925 (local address 127.0.98.2:37903)
0620 00:21:49.597936 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.597938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.598029 (+ 91us) server_negotiation.cc:407] Connection header received
0620 00:21:49.598075 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.598076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.598087 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.598154 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0620 00:21:49.598350 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.599129 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.600877 (+ 1748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.601481 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.601582 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.601596 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.601895 (+ 299us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.601949 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260620 00:21:49.602028 20558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.597786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:36501 (local address 127.0.98.1:40037)
0620 00:21:49.597919 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:49.597931 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:49.598008 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:49.598179 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:49.598180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:49.598183 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:49.598267 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.598270 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.599240 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.599241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.600718 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.600723 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.601650 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.601652 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.601780 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.601783 (+ 3us) client_negotiation.cc:769] Sending connection context
0620 00:21:49.601840 (+ 57us) client_negotiation.cc:240] Negotiation successful
0620 00:21:49.601902 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":178,"thread_start_us":61,"threads_started":1}
I20260620 00:21:49.602026 20559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.597767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:37903 (local address 127.0.98.1:35925)
0620 00:21:49.597951 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:49.597961 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:21:49.598017 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:21:49.598179 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:21:49.598180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:21:49.598182 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0620 00:21:49.598268 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.598271 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.599220 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.599221 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.600756 (+ 1535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:21:49.600758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.601652 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.601652 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:21:49.601761 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.601768 (+ 7us) client_negotiation.cc:769] Sending connection context
0620 00:21:49.601840 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:21:49.601902 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":84,"spinlock_wait_cycles":17664,"thread_start_us":90,"threads_started":1}
I20260620 00:21:49.602530 20489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ae5acc08cd43508585aac7c351a224" candidate_uuid: "72c4f432cd184494b0d49d786b22302b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" is_pre_election: true
I20260620 00:21:49.602762 20358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ae5acc08cd43508585aac7c351a224" candidate_uuid: "72c4f432cd184494b0d49d786b22302b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc465edf63ad4e69a0b8b9d02378f282" is_pre_election: true
I20260620 00:21:49.602938 20358 raft_consensus.cc:2454] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72c4f432cd184494b0d49d786b22302b in term 0.
I20260620 00:21:49.602928 20489 raft_consensus.cc:2454] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72c4f432cd184494b0d49d786b22302b in term 0.
I20260620 00:21:49.603240 20160 leader_election.cc:302] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c4f432cd184494b0d49d786b22302b, bc465edf63ad4e69a0b8b9d02378f282; no voters:
I20260620 00:21:49.603389 20557 raft_consensus.cc:2781] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260620 00:21:49.603446 20557 raft_consensus.cc:482] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260620 00:21:49.603477 20557 raft_consensus.cc:3037] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:49.604467 20557 raft_consensus.cc:504] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } }
I20260620 00:21:49.604676 20557 leader_election.cc:288] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [CANDIDATE]: Term 1 election: Requested vote from peers bc465edf63ad4e69a0b8b9d02378f282 (127.0.98.2:37903), ac95ff0c71d2449f8c2b201cbeb499e0 (127.0.98.3:36501)
I20260620 00:21:49.604892 20489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ae5acc08cd43508585aac7c351a224" candidate_uuid: "72c4f432cd184494b0d49d786b22302b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0"
I20260620 00:21:49.604893 20358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4ae5acc08cd43508585aac7c351a224" candidate_uuid: "72c4f432cd184494b0d49d786b22302b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bc465edf63ad4e69a0b8b9d02378f282"
I20260620 00:21:49.605006 20489 raft_consensus.cc:3037] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:49.604983 20358 raft_consensus.cc:3037] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:49.605850 20358 raft_consensus.cc:2454] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c4f432cd184494b0d49d786b22302b in term 1.
I20260620 00:21:49.605852 20489 raft_consensus.cc:2454] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72c4f432cd184494b0d49d786b22302b in term 1.
I20260620 00:21:49.606050 20160 leader_election.cc:302] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72c4f432cd184494b0d49d786b22302b, bc465edf63ad4e69a0b8b9d02378f282; no voters:
I20260620 00:21:49.606175 20557 raft_consensus.cc:2781] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:49.606343 20557 raft_consensus.cc:686] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 1 LEADER]: Becoming Leader. State: Replica: 72c4f432cd184494b0d49d786b22302b, State: Running, Role: LEADER
I20260620 00:21:49.606462 20557 consensus_queue.cc:237] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } }
I20260620 00:21:49.607427 20085 catalog_manager.cc:5495] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b reported cstate change: term changed from 0 to 1, leader changed from <none> to 72c4f432cd184494b0d49d786b22302b (127.0.98.1). New cstate: current_term: 1 leader_uuid: "72c4f432cd184494b0d49d786b22302b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72c4f432cd184494b0d49d786b22302b" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43525 } health_report { overall_health: HEALTHY } } }
I20260620 00:21:49.662856 20548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.660394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36532 (local address 127.0.98.1:43525)
0620 00:21:49.660447 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:49.660449 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:49.660455 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:21:49.660490 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:49.660491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:49.660512 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:49.660578 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:21:49.660797 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.661320 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.662167 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:49.662359 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:49.662415 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:49.662559 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:21:49.662673 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:21:49.662743 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:49.662778 (+ 35us) server_negotiation.cc:299] Negotiation successful
0620 00:21:49.662795 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:21:49.664857 20358 raft_consensus.cc:1261] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [term 1 FOLLOWER]: Refusing update from remote peer 72c4f432cd184494b0d49d786b22302b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:49.664881 20489 raft_consensus.cc:1261] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [term 1 FOLLOWER]: Refusing update from remote peer 72c4f432cd184494b0d49d786b22302b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:21:49.665237 20557 consensus_queue.cc:1035] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [LEADER]: Connected to new peer: Peer: permanent_uuid: "ac95ff0c71d2449f8c2b201cbeb499e0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 36501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:49.665377 20560 consensus_queue.cc:1035] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [LEADER]: Connected to new peer: Peer: permanent_uuid: "bc465edf63ad4e69a0b8b9d02378f282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 37903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:21:49.668370 20564 mvcc.cc:205] Tried to move back new op lower bound from 7298723469984002048 to 7298723469748838400. Current Snapshot: MvccSnapshot[applied={T|T < 7298723469984002048}]
I20260620 00:21:49.668430 20565 mvcc.cc:205] Tried to move back new op lower bound from 7298723469984002048 to 7298723469748838400. Current Snapshot: MvccSnapshot[applied={T|T < 7298723469984002048}]
I20260620 00:21:49.670301 20085 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:49.670393 20085 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37700:
table { table_name: "test-table" } modify_external_catalogs: true
I20260620 00:21:49.671185 20085 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0901b3bfd55f4897b07d8238306059fa: Sending DeleteTablet for 3 replicas of tablet a4ae5acc08cd43508585aac7c351a224
I20260620 00:21:49.671506 20338 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ae5acc08cd43508585aac7c351a224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:49 UTC) from {username='slave'} at 127.0.0.1:36464
I20260620 00:21:49.671579 20207 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ae5acc08cd43508585aac7c351a224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:49 UTC) from {username='slave'} at 127.0.0.1:36518
I20260620 00:21:49.671607 20469 tablet_service.cc:1514] Processing DeleteTablet for tablet a4ae5acc08cd43508585aac7c351a224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-06-20 00:21:49 UTC) from {username='slave'} at 127.0.0.1:47828
I20260620 00:21:49.671800 20573 tablet_replica.cc:330] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b: stopping tablet replica
I20260620 00:21:49.671833 20574 tablet_replica.cc:330] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: stopping tablet replica
I20260620 00:21:49.671940 20574 raft_consensus.cc:2229] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:49.671962 20573 raft_consensus.cc:2229] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 1 LEADER]: Raft consensus shutting down.
I20260620 00:21:49.672047 20574 raft_consensus.cc:2258] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:49.672092 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 20145
I20260620 00:21:49.672248 20573 raft_consensus.cc:2258] T a4ae5acc08cd43508585aac7c351a224 P 72c4f432cd184494b0d49d786b22302b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:49.672489 20574 ts_tablet_manager.cc:1905] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:49.673972 20574 ts_tablet_manager.cc:1918] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:49.674033 20574 log.cc:1196] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-2/wal/wals/a4ae5acc08cd43508585aac7c351a224
I20260620 00:21:49.674297 20574 ts_tablet_manager.cc:1939] T a4ae5acc08cd43508585aac7c351a224 P ac95ff0c71d2449f8c2b201cbeb499e0: Deleting consensus metadata
I20260620 00:21:49.674712 20070 catalog_manager.cc:4841] TS ac95ff0c71d2449f8c2b201cbeb499e0 (127.0.98.3:36501): tablet a4ae5acc08cd43508585aac7c351a224 (table test-table [id=3ac8d94198ad4a26a1e7736019dcb1e9]) successfully deleted
I20260620 00:21:49.676306 20572 tablet_replica.cc:330] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: stopping tablet replica
I20260620 00:21:49.676440 20572 raft_consensus.cc:2229] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260620 00:21:49.676582 20572 raft_consensus.cc:2258] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260620 00:21:49.677027 20572 ts_tablet_manager.cc:1905] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260620 00:21:49.678411 20572 ts_tablet_manager.cc:1918] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260620 00:21:49.678473 20572 log.cc:1196] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: Deleting WAL directory at /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1781914666918284-392-0/minicluster-data/ts-1/wal/wals/a4ae5acc08cd43508585aac7c351a224
I20260620 00:21:49.678702 20572 ts_tablet_manager.cc:1939] T a4ae5acc08cd43508585aac7c351a224 P bc465edf63ad4e69a0b8b9d02378f282: Deleting consensus metadata
W20260620 00:21:49.679100 20071 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43525 (error 108)
I20260620 00:21:49.679133 20069 catalog_manager.cc:4841] TS bc465edf63ad4e69a0b8b9d02378f282 (127.0.98.2:37903): tablet a4ae5acc08cd43508585aac7c351a224 (table test-table [id=3ac8d94198ad4a26a1e7736019dcb1e9]) successfully deleted
I20260620 00:21:49.679325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 20276
I20260620 00:21:49.680018 20543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:49.679450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43525 (local address 127.0.0.1:36538)
0620 00:21:49.679933 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0620 00:21:49.679981 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43525: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":460}
W20260620 00:21:49.680217 20071 catalog_manager.cc:4584] TS 72c4f432cd184494b0d49d786b22302b (127.0.98.1:43525): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a4ae5acc08cd43508585aac7c351a224: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43525: connect: Connection refused (error 111)
I20260620 00:21:49.684791 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 20407
I20260620 00:21:49.689690 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 20054
2026-06-20T00:21:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (990 ms)
[----------] 6 tests from EncryptionPolicyTest (9033 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:49 dist-test-slave-r81b krb5kdc[20578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:49 dist-test-slave-r81b krb5kdc[20578](info): set up 2 sockets
Jun 20 00:21:49 dist-test-slave-r81b krb5kdc[20578](info): commencing operation
krb5kdc: starting...
W20260620 00:21:50.744407 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:50 dist-test-slave-r81b krb5kdc[20578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:21:50.876250 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36551
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43257
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36551
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:50.968938 20591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:50.969188 20591 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:50.969214 20591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:50.970547 20591 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:50.970615 20591 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:50.970631 20591 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:50.970644 20591 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:50.970657 20591 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:50.972333 20591 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43257
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36551
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36551
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20591
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:50.972715 20591 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:50.972990 20591 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:50.978443 20596 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:50.978443 20599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:50.978466 20597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:50.978765 20591 server_base.cc:1034] running on GCE node
I20260620 00:21:50.979065 20591 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:50.979367 20591 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:50.980587 20591 hybrid_clock.cc:648] HybridClock initialized: now 1781914910980595 us; error 57 us; skew 500 ppm
Jun 20 00:21:50 dist-test-slave-r81b krb5kdc[20578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914910, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:50.983423 20591 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:21:50.983952 20591 webserver.cc:460] Webserver started at http://127.0.0.1:34451/ using document root <none> and password file <none>
I20260620 00:21:50.984217 20591 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:50.984266 20591 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:50.984371 20591 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:50.985494 20591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "29828f667339493283c120975122821e"
format_stamp: "Formatted at 2026-06-20 00:21:50 on dist-test-slave-r81b"
server_key: "880741ee55dad27f64a6e6da18b02523"
server_key_iv: "51a04ac8143cd1d9e50f4979790a9b73"
server_key_version: "encryptionkey@0"
I20260620 00:21:50.985918 20591 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "29828f667339493283c120975122821e"
format_stamp: "Formatted at 2026-06-20 00:21:50 on dist-test-slave-r81b"
server_key: "880741ee55dad27f64a6e6da18b02523"
server_key_iv: "51a04ac8143cd1d9e50f4979790a9b73"
server_key_version: "encryptionkey@0"
I20260620 00:21:50.987337 20591 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:21:50.988294 20606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:50.988464 20591 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:50.988564 20591 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "29828f667339493283c120975122821e"
format_stamp: "Formatted at 2026-06-20 00:21:50 on dist-test-slave-r81b"
server_key: "880741ee55dad27f64a6e6da18b02523"
server_key_iv: "51a04ac8143cd1d9e50f4979790a9b73"
server_key_version: "encryptionkey@0"
I20260620 00:21:50.988634 20591 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:51.003312 20591 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:51.006466 20591 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:51.006763 20591 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:51.011371 20591 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36551
I20260620 00:21:51.011409 20658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36551 every 8 connection(s)
I20260620 00:21:51.011782 20591 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:51.012185 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 20591
I20260620 00:21:51.012359 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:51.012612 392 external_mini_cluster.cc:1428] Setting key a22d6bc47ff0f8554e8cccf0329a0f09
I20260620 00:21:51.013155 20659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:51.017007 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e: Bootstrap starting.
I20260620 00:21:51.017799 20659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:51.018134 20659 log.cc:824] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:51.019197 20659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e: No bootstrap required, opened a new log
Jun 20 00:21:51 dist-test-slave-r81b krb5kdc[20578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914910, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:21:51.021811 20659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29828f667339493283c120975122821e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260620 00:21:51.022047 20659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:51.022090 20659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29828f667339493283c120975122821e, State: Initialized, Role: FOLLOWER
I20260620 00:21:51.022245 20659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29828f667339493283c120975122821e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260620 00:21:51.022334 20659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:51.022359 20659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:51.022469 20659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:51.023324 20659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29828f667339493283c120975122821e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260620 00:21:51.023453 20659 leader_election.cc:302] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 29828f667339493283c120975122821e; no voters:
I20260620 00:21:51.023725 20659 leader_election.cc:288] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:51.023787 20664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:51.023984 20664 raft_consensus.cc:686] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [term 1 LEADER]: Becoming Leader. State: Replica: 29828f667339493283c120975122821e, State: Running, Role: LEADER
I20260620 00:21:51.024080 20659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:51.024097 20664 consensus_queue.cc:237] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29828f667339493283c120975122821e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } }
I20260620 00:21:51.024557 20664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 29828f667339493283c120975122821e. Latest consensus state: current_term: 1 leader_uuid: "29828f667339493283c120975122821e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29828f667339493283c120975122821e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } } }
I20260620 00:21:51.024663 20664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:51.024569 20666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "29828f667339493283c120975122821e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "29828f667339493283c120975122821e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36551 } } }
I20260620 00:21:51.024911 20666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:51.025069 20673 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:51.025959 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:51.013227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55575 (local address 127.0.0.1:36551)
0620 00:21:51.013464 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:51.013475 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:51.013511 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:21:51.014134 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:51.014145 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:51.014219 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:51.014391 (+ 172us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:51.015033 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:51.016636 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:51.017533 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:51.018085 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:51.018222 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:51.021245 (+ 3023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:51.021252 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:51.021252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:51.021269 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:51.023519 (+ 2250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:51.024193 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:51.024198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:51.024198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:51.024321 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:51.024637 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:51.024638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:51.024639 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:51.025027 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:51.025174 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:51.025566 (+ 392us) server_negotiation.cc:299] Negotiation successful
0620 00:21:51.025706 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":111,"threads_started":1}
I20260620 00:21:51.026146 20673 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:51.027323 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:21:51.027870 20673 catalog_manager.cc:1347] Generated new cluster ID: 487dae9b5b69404ab8fe88978a1b8c70
I20260620 00:21:51.027926 20673 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:51.034343 20673 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:51.035112 20673 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:51.039672 20662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:51.032724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55577 (local address 127.0.0.1:36551)
0620 00:21:51.032779 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:51.032782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:51.033025 (+ 243us) server_negotiation.cc:407] Connection header received
0620 00:21:51.033102 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:51.033103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:51.033113 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:51.033162 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:51.033766 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:51.034458 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:51.035272 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:51.035483 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:51.035562 (+ 79us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:51.036951 (+ 1389us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:51.036952 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:51.036953 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:51.036961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:51.038427 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:51.038834 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:51.038835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:51.038836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:51.038893 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:51.039132 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:51.039132 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:51.039133 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:51.039249 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:51.039360 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:51.039494 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 00:21:51.039566 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260620 00:21:51.043565 20673 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 29828f667339493283c120975122821e: Generated new TSK 0
I20260620 00:21:51.043805 20673 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:51.047766 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 20591
W20260620 00:21:51.052232 20576 connection.cc:535] client connection to 127.0.0.1:36551 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36551): Connection reset by peer (error 104)
2026-06-20T00:21:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:51 dist-test-slave-r81b krb5kdc[20690](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:51 dist-test-slave-r81b krb5kdc[20690](info): set up 2 sockets
Jun 20 00:21:51 dist-test-slave-r81b krb5kdc[20690](info): commencing operation
krb5kdc: starting...
W20260620 00:21:52.116681 392 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:52 dist-test-slave-r81b krb5kdc[20690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914912, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:21:52.248392 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36465
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41737
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36465
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:52.333963 20703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:52.334192 20703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:52.334218 20703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:52.335351 20703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:52.335391 20703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:52.335405 20703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:52.335418 20703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:52.335430 20703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:52.336973 20703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41737
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36465
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36465
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:52.337201 20703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:52.337391 20703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:52.342517 20709 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:52.342525 20711 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:52.342525 20708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:52.342775 20703 server_base.cc:1034] running on GCE node
I20260620 00:21:52.343070 20703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:52.343438 20703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:52.344615 20703 hybrid_clock.cc:648] HybridClock initialized: now 1781914912344597 us; error 46 us; skew 500 ppm
Jun 20 00:21:52 dist-test-slave-r81b krb5kdc[20690](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914912, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:52.347417 20703 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:21:52.347946 20703 webserver.cc:460] Webserver started at http://127.0.0.1:43799/ using document root <none> and password file <none>
I20260620 00:21:52.348290 20703 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:52.348366 20703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:52.348527 20703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:52.349673 20703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "e4cf767b251642588ee735b7b8a37713"
format_stamp: "Formatted at 2026-06-20 00:21:52 on dist-test-slave-r81b"
server_key: "0aab4913817948e4d8dcc9348eec3948"
server_key_iv: "aedc1b333715de91738bd90501270851"
server_key_version: "encryptionkey@0"
I20260620 00:21:52.350064 20703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "e4cf767b251642588ee735b7b8a37713"
format_stamp: "Formatted at 2026-06-20 00:21:52 on dist-test-slave-r81b"
server_key: "0aab4913817948e4d8dcc9348eec3948"
server_key_iv: "aedc1b333715de91738bd90501270851"
server_key_version: "encryptionkey@0"
I20260620 00:21:52.351440 20703 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:21:52.352273 20718 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:52.352470 20703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:52.352576 20703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "e4cf767b251642588ee735b7b8a37713"
format_stamp: "Formatted at 2026-06-20 00:21:52 on dist-test-slave-r81b"
server_key: "0aab4913817948e4d8dcc9348eec3948"
server_key_iv: "aedc1b333715de91738bd90501270851"
server_key_version: "encryptionkey@0"
I20260620 00:21:52.352662 20703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:52.384253 20703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:52.386747 20703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:52.386926 20703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:52.391450 20703 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36465
I20260620 00:21:52.391644 20770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36465 every 8 connection(s)
I20260620 00:21:52.391944 20703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:52.392970 20771 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:52.394135 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 20703
I20260620 00:21:52.394279 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:52.394486 392 external_mini_cluster.cc:1428] Setting key 20816339ab5362cef2f6e31ea4c61362
I20260620 00:21:52.396279 20771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713: Bootstrap starting.
I20260620 00:21:52.397156 20771 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:52.397504 20771 log.cc:824] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:52.398402 20771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713: No bootstrap required, opened a new log
I20260620 00:21:52.401291 20771 raft_consensus.cc:348] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf767b251642588ee735b7b8a37713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260620 00:21:52.401559 20771 raft_consensus.cc:374] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:52.401652 20771 raft_consensus.cc:729] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e4cf767b251642588ee735b7b8a37713, State: Initialized, Role: FOLLOWER
I20260620 00:21:52.401808 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:52.395115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52933 (local address 127.0.0.1:36465)
0620 00:21:52.395350 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:52.395360 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:52.395399 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:21:52.396002 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:52.396012 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:52.396093 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:52.396301 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:52.396584 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:52.398148 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:52.399037 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:52.399748 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:52.399899 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:52.400932 (+ 1033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:52.400939 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:52.400939 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:52.401174 (+ 235us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:52.401219 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:52.401237 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:52.401343 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:52.401453 (+ 110us) server_negotiation.cc:299] Negotiation successful
0620 00:21:52.401485 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":109,"threads_started":1}
I20260620 00:21:52.401856 20771 consensus_queue.cc:260] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf767b251642588ee735b7b8a37713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260620 00:21:52.401947 20771 raft_consensus.cc:388] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:52.401998 20771 raft_consensus.cc:482] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:52.402045 20771 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:52.402781 20771 raft_consensus.cc:504] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf767b251642588ee735b7b8a37713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260620 00:21:52.402865 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:21:52.402925 20771 leader_election.cc:302] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e4cf767b251642588ee735b7b8a37713; no voters:
I20260620 00:21:52.403173 20771 leader_election.cc:288] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:52.403254 20776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:52.403445 20776 raft_consensus.cc:686] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [term 1 LEADER]: Becoming Leader. State: Replica: e4cf767b251642588ee735b7b8a37713, State: Running, Role: LEADER
I20260620 00:21:52.403476 20771 sys_catalog.cc:564] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:52.403564 20776 consensus_queue.cc:237] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf767b251642588ee735b7b8a37713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } }
I20260620 00:21:52.403970 20776 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e4cf767b251642588ee735b7b8a37713. Latest consensus state: current_term: 1 leader_uuid: "e4cf767b251642588ee735b7b8a37713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf767b251642588ee735b7b8a37713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } } }
I20260620 00:21:52.404227 20776 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:52.404122 20777 sys_catalog.cc:455] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e4cf767b251642588ee735b7b8a37713" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e4cf767b251642588ee735b7b8a37713" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36465 } } }
I20260620 00:21:52.404476 20777 sys_catalog.cc:458] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:52.404819 20788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:52.405316 20788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:52.407069 20788 catalog_manager.cc:1347] Generated new cluster ID: 03e67f8e28334e21902b2f0a0218a413
I20260620 00:21:52.407173 20788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:52.409032 20774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:52.405357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50681 (local address 127.0.0.1:36465)
0620 00:21:52.405404 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:52.405406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:52.405663 (+ 257us) server_negotiation.cc:407] Connection header received
0620 00:21:52.405757 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:52.405758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:52.405770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:52.405831 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:52.406053 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:52.406789 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:52.407633 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:52.407872 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:52.407969 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:52.408792 (+ 823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:52.408794 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:52.408794 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:52.408799 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:52.408813 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:52.408818 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:52.408873 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:52.408931 (+ 58us) server_negotiation.cc:299] Negotiation successful
0620 00:21:52.408941 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:21:52.414271 20788 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:52.414896 20788 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:52.419323 20788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e4cf767b251642588ee735b7b8a37713: Generated new TSK 0
I20260620 00:21:52.419535 20788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:52.422405 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 20703
W20260620 00:21:52.426762 20688 connection.cc:535] client connection to 127.0.0.1:36465 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36465): Connection reset by peer (error 104)
2026-06-20T00:21:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:52 dist-test-slave-r81b krb5kdc[20802](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:52 dist-test-slave-r81b krb5kdc[20802](info): set up 2 sockets
Jun 20 00:21:52 dist-test-slave-r81b krb5kdc[20802](info): commencing operation
krb5kdc: starting...
W20260620 00:21:53.480232 392 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:53 dist-test-slave-r81b krb5kdc[20802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914913, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:21:53.607358 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45281
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45927
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:53.689746 20815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:53.690013 20815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:53.690057 20815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:53.691250 20815 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:53.691329 20815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:53.691354 20815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:53.691382 20815 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:53.691412 20815 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:53.693053 20815 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45281
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45927
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45927
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:53.693403 20815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:53.693693 20815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:53.698952 20821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:53.699231 20815 server_base.cc:1034] running on GCE node
W20260620 00:21:53.698938 20823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:53.698938 20820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:53.699731 20815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:53.700062 20815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:53.701223 20815 hybrid_clock.cc:648] HybridClock initialized: now 1781914913701179 us; error 58 us; skew 500 ppm
Jun 20 00:21:53 dist-test-slave-r81b krb5kdc[20802](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914913, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:53.710513 20815 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:21:53.711099 20815 webserver.cc:460] Webserver started at http://127.0.0.1:35089/ using document root <none> and password file <none>
I20260620 00:21:53.711355 20815 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:53.711434 20815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:53.711623 20815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:53.712873 20815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "2630b31d218349fc86c5664faf5b8f46"
format_stamp: "Formatted at 2026-06-20 00:21:53 on dist-test-slave-r81b"
server_key: "f9c2b2bffde55cfe89c7f76ee47b5781"
server_key_iv: "e187a8b59935eb13ad6e49f73b894ffe"
server_key_version: "encryptionkey@0"
I20260620 00:21:53.713277 20815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "2630b31d218349fc86c5664faf5b8f46"
format_stamp: "Formatted at 2026-06-20 00:21:53 on dist-test-slave-r81b"
server_key: "f9c2b2bffde55cfe89c7f76ee47b5781"
server_key_iv: "e187a8b59935eb13ad6e49f73b894ffe"
server_key_version: "encryptionkey@0"
I20260620 00:21:53.714838 20815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:21:53.715742 20830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:53.716084 20815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:53.716246 20815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "2630b31d218349fc86c5664faf5b8f46"
format_stamp: "Formatted at 2026-06-20 00:21:53 on dist-test-slave-r81b"
server_key: "f9c2b2bffde55cfe89c7f76ee47b5781"
server_key_iv: "e187a8b59935eb13ad6e49f73b894ffe"
server_key_version: "encryptionkey@0"
I20260620 00:21:53.716332 20815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:53.719848 20815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:53.722292 20815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:53.722455 20815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:53.726576 20815 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45927
I20260620 00:21:53.726622 20882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45927 every 8 connection(s)
I20260620 00:21:53.726976 20815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:53.728055 20883 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:53.730429 20883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46: Bootstrap starting.
I20260620 00:21:53.731057 20883 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:53.731356 20883 log.cc:824] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:53.732097 20883 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46: No bootstrap required, opened a new log
I20260620 00:21:53.732708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 20815
I20260620 00:21:53.732834 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:53.733011 392 external_mini_cluster.cc:1428] Setting key d3e89895d7cf76d4a3eddd44ce517dab
I20260620 00:21:53.734362 20883 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2630b31d218349fc86c5664faf5b8f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260620 00:21:53.734556 20883 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:53.734694 20883 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2630b31d218349fc86c5664faf5b8f46, State: Initialized, Role: FOLLOWER
I20260620 00:21:53.734872 20883 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2630b31d218349fc86c5664faf5b8f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260620 00:21:53.734983 20883 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:53.735028 20883 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:53.735082 20883 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:53.735649 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:53.733595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56053 (local address 127.0.0.1:45927)
0620 00:21:53.733818 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:53.733830 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:53.733861 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:21:53.734336 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:53.734342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:53.734370 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:53.734490 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:53.735216 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:53.735220 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:53.735220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:53.735224 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:53.735270 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:53.735283 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:53.735363 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:53.735426 (+ 63us) server_negotiation.cc:299] Negotiation successful
0620 00:21:53.735468 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260620 00:21:53.736270 20883 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2630b31d218349fc86c5664faf5b8f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260620 00:21:53.736411 20883 leader_election.cc:302] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2630b31d218349fc86c5664faf5b8f46; no voters:
I20260620 00:21:53.736589 20883 leader_election.cc:288] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:53.736644 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:21:53.736629 20888 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:53.736790 20888 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [term 1 LEADER]: Becoming Leader. State: Replica: 2630b31d218349fc86c5664faf5b8f46, State: Running, Role: LEADER
I20260620 00:21:53.736894 20883 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:53.736950 20888 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2630b31d218349fc86c5664faf5b8f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } }
I20260620 00:21:53.737411 20890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2630b31d218349fc86c5664faf5b8f46. Latest consensus state: current_term: 1 leader_uuid: "2630b31d218349fc86c5664faf5b8f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2630b31d218349fc86c5664faf5b8f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260620 00:21:53.737421 20889 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2630b31d218349fc86c5664faf5b8f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2630b31d218349fc86c5664faf5b8f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45927 } } }
I20260620 00:21:53.737571 20889 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:53.737778 20890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46 [sys.catalog]: This master's current role is: LEADER
W20260620 00:21:53.738844 20908 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:21:53.738945 20908 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:21:53.739035 20897 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:53.739190 20897 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:53.739850 20887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:53.738656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37701 (local address 127.0.0.1:45927)
0620 00:21:53.738692 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:53.738694 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:53.738843 (+ 149us) server_negotiation.cc:407] Connection header received
0620 00:21:53.738922 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:53.738923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:53.738939 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:53.738983 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:53.739638 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:53.739639 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:53.739640 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:53.739644 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:53.739659 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:53.739663 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:53.739717 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:53.739769 (+ 52us) server_negotiation.cc:299] Negotiation successful
0620 00:21:53.739784 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260620 00:21:53.740805 20897 catalog_manager.cc:1347] Generated new cluster ID: c866f742acc444199c60afc1babda631
I20260620 00:21:53.740876 20897 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:53.757686 20897 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:53.757923 20897 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:53.760375 20897 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2630b31d218349fc86c5664faf5b8f46: Generated new TSK 0
I20260620 00:21:53.760609 20897 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:53.763276 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 20815
2026-06-20T00:21:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:53 dist-test-slave-r81b krb5kdc[20914](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:53 dist-test-slave-r81b krb5kdc[20914](info): set up 2 sockets
Jun 20 00:21:53 dist-test-slave-r81b krb5kdc[20914](info): commencing operation
krb5kdc: starting...
W20260620 00:21:54.876372 392 mini_kdc.cc:121] Time spent starting KDC: real 1.095s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:54 dist-test-slave-r81b krb5kdc[20914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:21:55.012874 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:55.097494 20927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:55.097733 20927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:55.097759 20927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:55.099160 20927 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:55.099220 20927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:55.099236 20927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:55.099249 20927 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:55.099264 20927 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:55.101002 20927 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45217
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:55.101234 20927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:55.101423 20927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:55.106678 20935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:55.106683 20933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:55.106678 20932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:55.106715 20927 server_base.cc:1034] running on GCE node
I20260620 00:21:55.107213 20927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:55.107561 20927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:55.108750 20927 hybrid_clock.cc:648] HybridClock initialized: now 1781914915108715 us; error 45 us; skew 500 ppm
Jun 20 00:21:55 dist-test-slave-r81b krb5kdc[20914](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914915, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:55.111925 20927 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:21:55.112576 20927 webserver.cc:460] Webserver started at http://127.0.0.1:39865/ using document root <none> and password file <none>
I20260620 00:21:55.112823 20927 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:55.112901 20927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:55.113059 20927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:55.114243 20927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "f338fad360604d8096a5529ecb3d5e0b"
format_stamp: "Formatted at 2026-06-20 00:21:55 on dist-test-slave-r81b"
server_key: "ce6aecbdcf533a86efa03f8e6996583f"
server_key_iv: "fa026f959726c46d309f551aca58ae44"
server_key_version: "encryptionkey@0"
I20260620 00:21:55.114653 20927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "f338fad360604d8096a5529ecb3d5e0b"
format_stamp: "Formatted at 2026-06-20 00:21:55 on dist-test-slave-r81b"
server_key: "ce6aecbdcf533a86efa03f8e6996583f"
server_key_iv: "fa026f959726c46d309f551aca58ae44"
server_key_version: "encryptionkey@0"
I20260620 00:21:55.116572 20927 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 00:21:55.118005 20942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:55.118221 20927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:55.118324 20927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "f338fad360604d8096a5529ecb3d5e0b"
format_stamp: "Formatted at 2026-06-20 00:21:55 on dist-test-slave-r81b"
server_key: "ce6aecbdcf533a86efa03f8e6996583f"
server_key_iv: "fa026f959726c46d309f551aca58ae44"
server_key_version: "encryptionkey@0"
I20260620 00:21:55.118413 20927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:55.136952 20927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:55.139501 20927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:55.139674 20927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:55.143661 20927 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45217
I20260620 00:21:55.143719 20994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45217 every 8 connection(s)
I20260620 00:21:55.144102 20927 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:55.145283 20995 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:55.148030 20995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b: Bootstrap starting.
I20260620 00:21:55.148232 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 20927
I20260620 00:21:55.148396 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:55.148574 392 external_mini_cluster.cc:1428] Setting key e440c697e57910acc58a15a443bc7215
I20260620 00:21:55.148895 20995 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:55.149299 20995 log.cc:824] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:55.150103 20995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b: No bootstrap required, opened a new log
I20260620 00:21:55.152530 20995 raft_consensus.cc:348] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f338fad360604d8096a5529ecb3d5e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260620 00:21:55.152742 20995 raft_consensus.cc:374] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:55.152779 20995 raft_consensus.cc:729] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f338fad360604d8096a5529ecb3d5e0b, State: Initialized, Role: FOLLOWER
I20260620 00:21:55.152906 20995 consensus_queue.cc:260] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f338fad360604d8096a5529ecb3d5e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260620 00:21:55.152978 20995 raft_consensus.cc:388] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:55.153007 20995 raft_consensus.cc:482] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:55.153044 20995 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:55.153694 20995 raft_consensus.cc:504] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f338fad360604d8096a5529ecb3d5e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260620 00:21:55.153834 20995 leader_election.cc:302] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f338fad360604d8096a5529ecb3d5e0b; no voters:
I20260620 00:21:55.154040 20995 leader_election.cc:288] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:55.154228 21001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:55.154353 20995 sys_catalog.cc:564] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:55.154353 21001 raft_consensus.cc:686] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [term 1 LEADER]: Becoming Leader. State: Replica: f338fad360604d8096a5529ecb3d5e0b, State: Running, Role: LEADER
I20260620 00:21:55.154486 21001 consensus_queue.cc:237] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f338fad360604d8096a5529ecb3d5e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } }
I20260620 00:21:55.154814 21001 sys_catalog.cc:455] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f338fad360604d8096a5529ecb3d5e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f338fad360604d8096a5529ecb3d5e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } } }
I20260620 00:21:55.154896 21001 sys_catalog.cc:458] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:55.155015 21001 sys_catalog.cc:455] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f338fad360604d8096a5529ecb3d5e0b. Latest consensus state: current_term: 1 leader_uuid: "f338fad360604d8096a5529ecb3d5e0b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f338fad360604d8096a5529ecb3d5e0b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45217 } } }
I20260620 00:21:55.155086 21001 sys_catalog.cc:458] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:55.155221 21007 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:55.155750 21007 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Jun 20 00:21:55 dist-test-slave-r81b krb5kdc[20914](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914914, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:21:55.157400 21017 mvcc.cc:205] Tried to move back new op lower bound from 7298723492482437120 to 7298723492473069568. Current Snapshot: MvccSnapshot[applied={T|T < 7298723492482437120}]
I20260620 00:21:55.157864 21007 catalog_manager.cc:1347] Generated new cluster ID: f5ecc4573f6e467095f71eacec452937
I20260620 00:21:55.157909 21007 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:55.160779 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:55.149218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39969 (local address 127.0.0.1:45217)
0620 00:21:55.149433 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:55.149444 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:55.149479 (+ 35us) server_negotiation.cc:407] Connection header received
0620 00:21:55.150006 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:55.150014 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:55.150080 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:55.150270 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:55.151117 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:55.152116 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:55.153232 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:55.153661 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:55.153750 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:55.156813 (+ 3063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:55.156818 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:55.156818 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:55.156831 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:55.158952 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:55.159426 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:55.159430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:55.159430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:55.159550 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:55.159772 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:55.159773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:55.159773 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:55.160084 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:55.160271 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:55.160405 (+ 134us) server_negotiation.cc:299] Negotiation successful
0620 00:21:55.160554 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260620 00:21:55.161783 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:21:55.169523 21007 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:55.170280 21007 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:55.170871 20998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:55.163373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58823 (local address 127.0.0.1:45217)
0620 00:21:55.163425 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:55.163426 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:55.163535 (+ 109us) server_negotiation.cc:407] Connection header received
0620 00:21:55.163568 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:55.163569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:55.163580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:55.163652 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:55.164287 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:55.165087 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:55.166092 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:55.166411 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:55.166487 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:55.167907 (+ 1420us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:55.167909 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:55.167909 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:55.167917 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:55.169610 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:55.170009 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:55.170010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:55.170010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:55.170066 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:55.170279 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:55.170279 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:55.170280 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:55.170429 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:55.170521 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:55.170692 (+ 171us) server_negotiation.cc:299] Negotiation successful
0620 00:21:55.170785 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:21:55.174978 21007 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f338fad360604d8096a5529ecb3d5e0b: Generated new TSK 0
I20260620 00:21:55.175196 21007 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:55.179097 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 20927
2026-06-20T00:21:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1415 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:55 dist-test-slave-r81b krb5kdc[21026](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:55 dist-test-slave-r81b krb5kdc[21026](info): set up 2 sockets
Jun 20 00:21:55 dist-test-slave-r81b krb5kdc[21026](info): commencing operation
krb5kdc: starting...
W20260620 00:21:56.248272 392 mini_kdc.cc:121] Time spent starting KDC: real 1.051s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:56 dist-test-slave-r81b krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914916, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:21:56.375315 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44701
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44701
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:56.458935 21039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:56.459195 21039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:56.459233 21039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:56.460446 21039 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:56.460531 21039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:56.460561 21039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:56.460592 21039 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:56.460614 21039 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:56.462109 21039 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44701
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44701
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:56.462392 21039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:56.462642 21039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:56.468665 21044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:56.468822 21047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:56.468693 21045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:56.468746 21039 server_base.cc:1034] running on GCE node
I20260620 00:21:56.469221 21039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:56.469537 21039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:56.470719 21039 hybrid_clock.cc:648] HybridClock initialized: now 1781914916470723 us; error 69 us; skew 500 ppm
Jun 20 00:21:56 dist-test-slave-r81b krb5kdc[21026](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914916, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:56.473618 21039 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:21:56.474115 21039 webserver.cc:460] Webserver started at http://127.0.0.1:33717/ using document root <none> and password file <none>
I20260620 00:21:56.474385 21039 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:56.474453 21039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:56.474618 21039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:56.475780 21039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "f85bfeccaaf94867b15e3a655c9537ba"
format_stamp: "Formatted at 2026-06-20 00:21:56 on dist-test-slave-r81b"
server_key: "9c44ce143f5bfd9d788b410028fc6488"
server_key_iv: "b75c1872bf37da259ce004218266d07c"
server_key_version: "encryptionkey@0"
I20260620 00:21:56.476197 21039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "f85bfeccaaf94867b15e3a655c9537ba"
format_stamp: "Formatted at 2026-06-20 00:21:56 on dist-test-slave-r81b"
server_key: "9c44ce143f5bfd9d788b410028fc6488"
server_key_iv: "b75c1872bf37da259ce004218266d07c"
server_key_version: "encryptionkey@0"
I20260620 00:21:56.477617 21039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:21:56.478480 21054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:56.478713 21039 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:56.478821 21039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "f85bfeccaaf94867b15e3a655c9537ba"
format_stamp: "Formatted at 2026-06-20 00:21:56 on dist-test-slave-r81b"
server_key: "9c44ce143f5bfd9d788b410028fc6488"
server_key_iv: "b75c1872bf37da259ce004218266d07c"
server_key_version: "encryptionkey@0"
I20260620 00:21:56.478910 21039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:56.508858 21039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:56.510969 21039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:56.511157 21039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:56.515463 21039 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44701
I20260620 00:21:56.515545 21106 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44701 every 8 connection(s)
I20260620 00:21:56.515949 21039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:56.516841 21107 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:56.519186 21107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba: Bootstrap starting.
I20260620 00:21:56.519786 21107 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:56.520083 21107 log.cc:824] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:56.520881 21107 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba: No bootstrap required, opened a new log
I20260620 00:21:56.521010 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 21039
I20260620 00:21:56.521131 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:56.521332 392 external_mini_cluster.cc:1428] Setting key b66ee43e1571d7b752a16b2a02d64ea2
I20260620 00:21:56.523373 21107 raft_consensus.cc:348] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85bfeccaaf94867b15e3a655c9537ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } }
I20260620 00:21:56.523562 21107 raft_consensus.cc:374] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:56.523603 21107 raft_consensus.cc:729] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f85bfeccaaf94867b15e3a655c9537ba, State: Initialized, Role: FOLLOWER
I20260620 00:21:56.523761 21107 consensus_queue.cc:260] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85bfeccaaf94867b15e3a655c9537ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } }
I20260620 00:21:56.523839 21107 raft_consensus.cc:388] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:56.523869 21107 raft_consensus.cc:482] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:56.523916 21107 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:56.524696 21107 raft_consensus.cc:504] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85bfeccaaf94867b15e3a655c9537ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } }
I20260620 00:21:56.524806 21107 leader_election.cc:302] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f85bfeccaaf94867b15e3a655c9537ba; no voters:
I20260620 00:21:56.524955 21107 leader_election.cc:288] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:56.525017 21112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:56.525214 21107 sys_catalog.cc:564] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:56.525213 21112 raft_consensus.cc:686] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [term 1 LEADER]: Becoming Leader. State: Replica: f85bfeccaaf94867b15e3a655c9537ba, State: Running, Role: LEADER
I20260620 00:21:56.525432 21112 consensus_queue.cc:237] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85bfeccaaf94867b15e3a655c9537ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } }
I20260620 00:21:56.525821 21113 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f85bfeccaaf94867b15e3a655c9537ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85bfeccaaf94867b15e3a655c9537ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } } }
I20260620 00:21:56.526033 21113 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:56.525965 21114 sys_catalog.cc:455] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader f85bfeccaaf94867b15e3a655c9537ba. Latest consensus state: current_term: 1 leader_uuid: "f85bfeccaaf94867b15e3a655c9537ba" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f85bfeccaaf94867b15e3a655c9537ba" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44701 } } }
I20260620 00:21:56.526268 21114 sys_catalog.cc:458] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:56.526423 21117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:56.526970 21117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:56.528604 21117 catalog_manager.cc:1347] Generated new cluster ID: cda122d6807a49fca31bb4a238b936bd
I20260620 00:21:56.528671 21117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:56.528769 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:56.521880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45825 (local address 127.0.0.1:44701)
0620 00:21:56.522089 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:56.522099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:56.522133 (+ 34us) server_negotiation.cc:407] Connection header received
0620 00:21:56.522897 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:56.522908 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:56.522976 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:56.523144 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:56.523373 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:56.525074 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:56.526301 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:56.526994 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:56.527734 (+ 740us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:56.527921 (+ 187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:56.527925 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:56.527925 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:56.528202 (+ 277us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:56.528241 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:56.528256 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:56.528344 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:56.528423 (+ 79us) server_negotiation.cc:299] Negotiation successful
0620 00:21:56.528492 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":93,"threads_started":1}
I20260620 00:21:56.533268 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:21:56.541640 21111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:56.534866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47749 (local address 127.0.0.1:44701)
0620 00:21:56.534907 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:56.534909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:56.538716 (+ 3807us) server_negotiation.cc:407] Connection header received
0620 00:21:56.538798 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:56.538799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:56.538810 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:56.538890 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:56.539056 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:56.539540 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:56.540351 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:56.540555 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:56.540630 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:56.541361 (+ 731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:56.541362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:56.541363 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:56.541367 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:56.541385 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:56.541389 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:56.541455 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:56.541516 (+ 61us) server_negotiation.cc:299] Negotiation successful
0620 00:21:56.541540 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:21:56.563935 21117 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:56.564761 21117 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:56.569676 21117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f85bfeccaaf94867b15e3a655c9537ba: Generated new TSK 0
I20260620 00:21:56.569886 21117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:56.579578 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 21039
2026-06-20T00:21:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:56 dist-test-slave-r81b krb5kdc[21138](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:56 dist-test-slave-r81b krb5kdc[21138](info): set up 2 sockets
Jun 20 00:21:56 dist-test-slave-r81b krb5kdc[21138](info): commencing operation
krb5kdc: starting...
W20260620 00:21:57.630062 392 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:57 dist-test-slave-r81b krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914917, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:21:57.763856 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39501
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:57.848613 21151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:57.848893 21151 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:57.848937 21151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:57.850227 21151 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:57.850299 21151 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:57.850322 21151 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:57.850344 21151 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:57.850374 21151 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:57.851912 21151 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21151
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:57.852231 21151 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:57.852478 21151 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:57.857513 21156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:57.857525 21157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:57.857688 21151 server_base.cc:1034] running on GCE node
W20260620 00:21:57.857617 21159 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:57.857990 21151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:57.858321 21151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:57.859495 21151 hybrid_clock.cc:648] HybridClock initialized: now 1781914917859496 us; error 65 us; skew 500 ppm
Jun 20 00:21:57 dist-test-slave-r81b krb5kdc[21138](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914917, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:57.862232 21151 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:21:57.862720 21151 webserver.cc:460] Webserver started at http://127.0.0.1:40727/ using document root <none> and password file <none>
I20260620 00:21:57.863005 21151 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:57.863088 21151 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:57.863262 21151 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:57.864516 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "4b20a815d43c4559ba8c7b2b44050f59"
format_stamp: "Formatted at 2026-06-20 00:21:57 on dist-test-slave-r81b"
server_key: "b7a531e060f54bd83c4cf7decd5e64b8"
server_key_iv: "8c3cab27bda0518f0dcbd7508d32971d"
server_key_version: "encryptionkey@0"
I20260620 00:21:57.864892 21151 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b20a815d43c4559ba8c7b2b44050f59"
format_stamp: "Formatted at 2026-06-20 00:21:57 on dist-test-slave-r81b"
server_key: "b7a531e060f54bd83c4cf7decd5e64b8"
server_key_iv: "8c3cab27bda0518f0dcbd7508d32971d"
server_key_version: "encryptionkey@0"
I20260620 00:21:57.866335 21151 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:57.867372 21166 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:57.867813 21151 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:21:57.867952 21151 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "4b20a815d43c4559ba8c7b2b44050f59"
format_stamp: "Formatted at 2026-06-20 00:21:57 on dist-test-slave-r81b"
server_key: "b7a531e060f54bd83c4cf7decd5e64b8"
server_key_iv: "8c3cab27bda0518f0dcbd7508d32971d"
server_key_version: "encryptionkey@0"
I20260620 00:21:57.868050 21151 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:57.871853 21151 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:57.874099 21151 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:57.874287 21151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:57.878756 21151 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39501
I20260620 00:21:57.878798 21218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39501 every 8 connection(s)
I20260620 00:21:57.879220 21151 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:57.880503 21219 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:57.882880 21219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59: Bootstrap starting.
I20260620 00:21:57.883458 21219 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:57.883709 21219 log.cc:824] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:57.884562 21219 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59: No bootstrap required, opened a new log
I20260620 00:21:57.886225 21219 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b20a815d43c4559ba8c7b2b44050f59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } }
I20260620 00:21:57.886379 21219 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:57.886402 21219 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b20a815d43c4559ba8c7b2b44050f59, State: Initialized, Role: FOLLOWER
I20260620 00:21:57.886555 21219 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b20a815d43c4559ba8c7b2b44050f59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } }
I20260620 00:21:57.886641 21219 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:57.886679 21219 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:57.886716 21219 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:57.887260 21219 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b20a815d43c4559ba8c7b2b44050f59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } }
I20260620 00:21:57.887347 21219 leader_election.cc:302] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b20a815d43c4559ba8c7b2b44050f59; no voters:
I20260620 00:21:57.887490 21219 leader_election.cc:288] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:57.887578 21222 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:57.887786 21222 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [term 1 LEADER]: Becoming Leader. State: Replica: 4b20a815d43c4559ba8c7b2b44050f59, State: Running, Role: LEADER
I20260620 00:21:57.887820 21219 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:57.887948 21222 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b20a815d43c4559ba8c7b2b44050f59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } }
I20260620 00:21:57.888404 21223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b20a815d43c4559ba8c7b2b44050f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b20a815d43c4559ba8c7b2b44050f59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } } }
I20260620 00:21:57.888495 21223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:57.888545 21224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b20a815d43c4559ba8c7b2b44050f59. Latest consensus state: current_term: 1 leader_uuid: "4b20a815d43c4559ba8c7b2b44050f59" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b20a815d43c4559ba8c7b2b44050f59" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39501 } } }
I20260620 00:21:57.888614 21224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:57.888741 21230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:57.889091 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 21151
I20260620 00:21:57.889250 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:57.889356 21230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:57.889451 392 external_mini_cluster.cc:1428] Setting key 9d8f1bca4adf61f21666ddf4e7744e92
I20260620 00:21:57.891438 21230 catalog_manager.cc:1347] Generated new cluster ID: b9ddc47e90854fc2874a41d3313de282
I20260620 00:21:57.891494 21230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:21:57.891958 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:57.890096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51343 (local address 127.0.0.1:39501)
0620 00:21:57.890273 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:57.890282 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:57.890314 (+ 32us) server_negotiation.cc:407] Connection header received
0620 00:21:57.890436 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:57.890443 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:57.890477 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:57.890625 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:57.891426 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:57.891430 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:57.891430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:57.891434 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:57.891469 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:57.891483 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:57.891702 (+ 219us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:57.891734 (+ 32us) server_negotiation.cc:299] Negotiation successful
0620 00:21:57.891781 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260620 00:21:57.892941 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:21:57.895663 21241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:57.894634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53817 (local address 127.0.0.1:39501)
0620 00:21:57.894667 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:57.894668 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:57.894676 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:21:57.894700 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:57.894701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:57.894705 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:57.894753 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:57.895417 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:57.895419 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:57.895419 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:21:57.895423 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:57.895438 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:21:57.895442 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:57.895496 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:57.895570 (+ 74us) server_negotiation.cc:299] Negotiation successful
0620 00:21:57.895583 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260620 00:21:57.933105 21230 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:57.933429 21230 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:57.937582 21230 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b20a815d43c4559ba8c7b2b44050f59: Generated new TSK 0
I20260620 00:21:57.937790 21230 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:57.947063 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 21151
2026-06-20T00:21:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1367 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:58 dist-test-slave-r81b krb5kdc[21250](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:58 dist-test-slave-r81b krb5kdc[21250](info): set up 2 sockets
Jun 20 00:21:58 dist-test-slave-r81b krb5kdc[21250](info): commencing operation
krb5kdc: starting...
W20260620 00:21:59.000592 392 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:21:59 dist-test-slave-r81b krb5kdc[21250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:21:59.136603 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44977
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44977
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:21:59.224557 21263 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:21:59.224865 21263 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:21:59.224910 21263 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:21:59.226153 21263 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:21:59.226233 21263 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:21:59.226259 21263 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:21:59.226287 21263 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:21:59.226320 21263 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:21:59.227869 21263 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44977
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44977
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21263
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:21:59.228186 21263 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:21:59.228432 21263 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:21:59.233593 21271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:21:59.233599 21269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:59.233618 21263 server_base.cc:1034] running on GCE node
W20260620 00:21:59.233608 21268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:21:59.234033 21263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:21:59.234369 21263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:21:59.235548 21263 hybrid_clock.cc:648] HybridClock initialized: now 1781914919235533 us; error 28 us; skew 500 ppm
Jun 20 00:21:59 dist-test-slave-r81b krb5kdc[21250](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:21:59.242551 21263 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:21:59.243034 21263 webserver.cc:460] Webserver started at http://127.0.0.1:35899/ using document root <none> and password file <none>
I20260620 00:21:59.243286 21263 fs_manager.cc:359] Metadata directory not provided
I20260620 00:21:59.243356 21263 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:21:59.243508 21263 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:21:59.244674 21263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "9239230cc8e04dd598f7d026929b3096"
format_stamp: "Formatted at 2026-06-20 00:21:59 on dist-test-slave-r81b"
server_key: "6a98dc9e304c7f26dd1a3f3df3361b49"
server_key_iv: "a66938c55ed5269742bbd603298cab61"
server_key_version: "encryptionkey@0"
I20260620 00:21:59.245035 21263 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "9239230cc8e04dd598f7d026929b3096"
format_stamp: "Formatted at 2026-06-20 00:21:59 on dist-test-slave-r81b"
server_key: "6a98dc9e304c7f26dd1a3f3df3361b49"
server_key_iv: "a66938c55ed5269742bbd603298cab61"
server_key_version: "encryptionkey@0"
I20260620 00:21:59.246407 21263 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:21:59.247143 21278 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:21:59.247336 21263 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260620 00:21:59.247447 21263 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "9239230cc8e04dd598f7d026929b3096"
format_stamp: "Formatted at 2026-06-20 00:21:59 on dist-test-slave-r81b"
server_key: "6a98dc9e304c7f26dd1a3f3df3361b49"
server_key_iv: "a66938c55ed5269742bbd603298cab61"
server_key_version: "encryptionkey@0"
I20260620 00:21:59.247522 21263 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:21:59.268052 21263 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:21:59.270987 21263 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:21:59.271195 21263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:21:59.275463 21263 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44977
I20260620 00:21:59.275519 21330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44977 every 8 connection(s)
I20260620 00:21:59.275925 21263 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:21:59.276862 21331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:21:59.279238 21331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096: Bootstrap starting.
I20260620 00:21:59.280071 21331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096: Neither blocks nor log segments found. Creating new log.
I20260620 00:21:59.280442 21331 log.cc:824] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096: Log is configured to *not* fsync() on all Append() calls
I20260620 00:21:59.281211 21331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096: No bootstrap required, opened a new log
I20260620 00:21:59.282879 21331 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9239230cc8e04dd598f7d026929b3096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260620 00:21:59.283015 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 21263
I20260620 00:21:59.283061 21331 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:21:59.283113 21331 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9239230cc8e04dd598f7d026929b3096, State: Initialized, Role: FOLLOWER
I20260620 00:21:59.283157 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:21:59.283326 392 external_mini_cluster.cc:1428] Setting key 40b2f6b41a66550cf7301517d91c3163
I20260620 00:21:59.283309 21331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9239230cc8e04dd598f7d026929b3096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260620 00:21:59.283403 21331 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:21:59.283443 21331 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:21:59.283509 21331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:21:59.284761 21331 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9239230cc8e04dd598f7d026929b3096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260620 00:21:59.284859 21331 leader_election.cc:302] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9239230cc8e04dd598f7d026929b3096; no voters:
I20260620 00:21:59.285039 21331 leader_election.cc:288] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:21:59.285094 21336 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:21:59.285239 21336 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [term 1 LEADER]: Becoming Leader. State: Replica: 9239230cc8e04dd598f7d026929b3096, State: Running, Role: LEADER
I20260620 00:21:59.285377 21331 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:21:59.285351 21336 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9239230cc8e04dd598f7d026929b3096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } }
I20260620 00:21:59.285751 21336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9239230cc8e04dd598f7d026929b3096. Latest consensus state: current_term: 1 leader_uuid: "9239230cc8e04dd598f7d026929b3096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9239230cc8e04dd598f7d026929b3096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } } }
I20260620 00:21:59.285835 21336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:59.285983 21338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9239230cc8e04dd598f7d026929b3096" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9239230cc8e04dd598f7d026929b3096" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44977 } } }
I20260620 00:21:59.286064 21338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096 [sys.catalog]: This master's current role is: LEADER
I20260620 00:21:59.286295 21340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:21:59.286871 21340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:21:59.288587 21340 catalog_manager.cc:1347] Generated new cluster ID: 3606a8865aed4ef1aa8f954a47f35837
I20260620 00:21:59.288643 21340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:21:59 dist-test-slave-r81b krb5kdc[21250](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:21:59.295257 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:59.283857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:51643 (local address 127.0.0.1:44977)
0620 00:21:59.284080 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:59.284095 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:59.284152 (+ 57us) server_negotiation.cc:407] Connection header received
0620 00:21:59.284322 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:59.284332 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:59.284410 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:59.284562 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:59.285444 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:59.287008 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:59.288174 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:59.288822 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:59.288957 (+ 135us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:59.291419 (+ 2462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:59.291423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:59.291423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:59.291436 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:59.293278 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:59.293890 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:59.293894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:59.293895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:59.294024 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:59.294284 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:59.294285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:59.294286 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:59.294581 (+ 295us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:59.294791 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:59.294908 (+ 117us) server_negotiation.cc:299] Negotiation successful
0620 00:21:59.295026 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":108,"threads_started":1}
I20260620 00:21:59.296295 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:21:59.300073 21340 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:21:59.300823 21340 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:21:59.304382 21335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:21:59.297703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:47325 (local address 127.0.0.1:44977)
0620 00:21:59.297780 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0620 00:21:59.297782 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:21:59.297831 (+ 49us) server_negotiation.cc:407] Connection header received
0620 00:21:59.297906 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:21:59.297907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:21:59.297916 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:21:59.297960 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:21:59.298509 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:59.299025 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:59.299862 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:21:59.300197 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:21:59.300260 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:21:59.301617 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:21:59.301619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:21:59.301619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:21:59.301627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:21:59.303121 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:59.303515 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:59.303517 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:59.303517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:59.303583 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:21:59.303811 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:21:59.303812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:21:59.303812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:21:59.303930 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:21:59.304045 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0620 00:21:59.304212 (+ 167us) server_negotiation.cc:299] Negotiation successful
0620 00:21:59.304279 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260620 00:21:59.312985 21340 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9239230cc8e04dd598f7d026929b3096: Generated new TSK 0
I20260620 00:21:59.313252 21340 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:21:59.319128 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 21263
W20260620 00:21:59.323648 21248 connection.cc:535] client connection to 127.0.0.1:44977 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44977): Connection reset by peer (error 104)
2026-06-20T00:21:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1372 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:21:59 dist-test-slave-r81b krb5kdc[21362](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:21:59 dist-test-slave-r81b krb5kdc[21362](info): set up 2 sockets
Jun 20 00:21:59 dist-test-slave-r81b krb5kdc[21362](info): commencing operation
krb5kdc: starting...
W20260620 00:22:00.368575 392 mini_kdc.cc:121] Time spent starting KDC: real 1.031s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:00 dist-test-slave-r81b krb5kdc[21362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914920, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:22:00.495589 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:00.580979 21375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:00.581218 21375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:00.581243 21375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:00.582465 21375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:00.582518 21375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:00.582533 21375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:00.582547 21375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:00.582559 21375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:00.584084 21375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21375
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:00.584400 21375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:00.584621 21375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:00.591419 21380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:00.591431 21375 server_base.cc:1034] running on GCE node
W20260620 00:22:00.591454 21383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:00.591578 21381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:00.592272 21375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:00.592633 21375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:00.593852 21375 hybrid_clock.cc:648] HybridClock initialized: now 1781914920593853 us; error 49 us; skew 500 ppm
Jun 20 00:22:00 dist-test-slave-r81b krb5kdc[21362](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:00.597126 21375 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:22:00.597882 21375 webserver.cc:460] Webserver started at http://127.0.0.1:43149/ using document root <none> and password file <none>
I20260620 00:22:00.598186 21375 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:00.598268 21375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:00.598443 21375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:00.599802 21375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "3202a1572cd942a6bbcf63fd1a2dea05"
format_stamp: "Formatted at 2026-06-20 00:22:00 on dist-test-slave-r81b"
server_key: "44fb04d687e7a4475a02c2097a7d653b"
server_key_iv: "b1b569effcb2e252af62fc8461e68274"
server_key_version: "encryptionkey@0"
I20260620 00:22:00.600332 21375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "3202a1572cd942a6bbcf63fd1a2dea05"
format_stamp: "Formatted at 2026-06-20 00:22:00 on dist-test-slave-r81b"
server_key: "44fb04d687e7a4475a02c2097a7d653b"
server_key_iv: "b1b569effcb2e252af62fc8461e68274"
server_key_version: "encryptionkey@0"
I20260620 00:22:00.602393 21375 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260620 00:22:00.603554 21390 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:00.603911 21375 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:00.604049 21375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "3202a1572cd942a6bbcf63fd1a2dea05"
format_stamp: "Formatted at 2026-06-20 00:22:00 on dist-test-slave-r81b"
server_key: "44fb04d687e7a4475a02c2097a7d653b"
server_key_iv: "b1b569effcb2e252af62fc8461e68274"
server_key_version: "encryptionkey@0"
I20260620 00:22:00.604619 21375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:00.618364 21375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:00.621650 21375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:00.621876 21375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:00.627213 21375 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41703
I20260620 00:22:00.627272 21442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41703 every 8 connection(s)
I20260620 00:22:00.627688 21375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:00.628636 21443 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:00.631230 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05: Bootstrap starting.
I20260620 00:22:00.631333 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 21375
I20260620 00:22:00.631443 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:00.631644 392 external_mini_cluster.cc:1428] Setting key 6ed12efcadcd8e6d7028e82350574f11
I20260620 00:22:00.632030 21443 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:00.632450 21443 log.cc:824] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:00.633208 21443 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05: No bootstrap required, opened a new log
I20260620 00:22:00.635040 21443 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3202a1572cd942a6bbcf63fd1a2dea05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41703 } }
I20260620 00:22:00.635226 21443 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:00.635275 21443 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3202a1572cd942a6bbcf63fd1a2dea05, State: Initialized, Role: FOLLOWER
I20260620 00:22:00.635421 21443 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3202a1572cd942a6bbcf63fd1a2dea05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41703 } }
I20260620 00:22:00.635504 21443 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:00.635564 21443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:00.635620 21443 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:00.636284 21443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3202a1572cd942a6bbcf63fd1a2dea05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41703 } }
I20260620 00:22:00.636408 21443 leader_election.cc:302] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3202a1572cd942a6bbcf63fd1a2dea05; no voters:
I20260620 00:22:00.636596 21443 leader_election.cc:288] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:00.636648 21448 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:00.636889 21443 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:00.637054 21448 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [term 1 LEADER]: Becoming Leader. State: Replica: 3202a1572cd942a6bbcf63fd1a2dea05, State: Running, Role: LEADER
I20260620 00:22:00.637161 21448 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3202a1572cd942a6bbcf63fd1a2dea05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41703 } }
I20260620 00:22:00.637544 21448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3202a1572cd942a6bbcf63fd1a2dea05. Latest consensus state: current_term: 1 leader_uuid: "3202a1572cd942a6bbcf63fd1a2dea05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3202a1572cd942a6bbcf63fd1a2dea05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41703 } } }
I20260620 00:22:00.637637 21448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:00.637691 21449 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3202a1572cd942a6bbcf63fd1a2dea05" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3202a1572cd942a6bbcf63fd1a2dea05" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41703 } } }
I20260620 00:22:00.637775 21449 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:00.638084 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:00.632495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:56269 (local address 127.0.0.1:41703)
0620 00:22:00.632699 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:00.632716 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:00.632743 (+ 27us) server_negotiation.cc:407] Connection header received
0620 00:22:00.633206 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:00.633212 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:00.633263 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:00.633457 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:00.633661 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:00.634573 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:00.635570 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:00.636076 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:00.636230 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:00.636991 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:00.636995 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:00.636996 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:00.637210 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:00.637252 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:00.637267 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:00.637631 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:00.637681 (+ 50us) server_negotiation.cc:299] Negotiation successful
0620 00:22:00.637713 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260620 00:22:00.638145 21456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:00.638697 21456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:00.639159 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:00.640316 21456 catalog_manager.cc:1347] Generated new cluster ID: 52ad82aaafdc4652a778f24b6ec5e6ce
I20260620 00:22:00.640460 21456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:00.644572 21446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:00.640684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:48917 (local address 127.0.0.1:41703)
0620 00:22:00.640720 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:00.640722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:00.640851 (+ 129us) server_negotiation.cc:407] Connection header received
0620 00:22:00.640947 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:00.640948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:00.640959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:00.640988 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:00.641293 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:00.642065 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:00.643130 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:00.643453 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:00.643516 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:00.644305 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:00.644307 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:00.644307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:00.644312 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:00.644328 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:00.644332 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:00.644382 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:00.644481 (+ 99us) server_negotiation.cc:299] Negotiation successful
0620 00:22:00.644491 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 00:22:00.664664 21456 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:00.665445 21456 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:00.671806 21456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3202a1572cd942a6bbcf63fd1a2dea05: Generated new TSK 0
I20260620 00:22:00.672127 21456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:00.678081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 21375
W20260620 00:22:00.682870 21360 connection.cc:535] client connection to 127.0.0.1:41703 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41703): Connection reset by peer (error 104)
2026-06-20T00:22:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:00 dist-test-slave-r81b krb5kdc[21474](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:00 dist-test-slave-r81b krb5kdc[21474](info): set up 2 sockets
Jun 20 00:22:00 dist-test-slave-r81b krb5kdc[21474](info): commencing operation
krb5kdc: starting...
W20260620 00:22:01.744396 392 mini_kdc.cc:121] Time spent starting KDC: real 1.046s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:01 dist-test-slave-r81b krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914921, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:22:01.871706 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40847
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38467
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:01.955507 21487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:01.955755 21487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:01.955782 21487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:01.957140 21487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:01.957192 21487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:01.957207 21487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:01.957221 21487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:01.957232 21487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:01.958820 21487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38467
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:01.959096 21487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:01.959357 21487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:01.964428 21493 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:01.964573 21495 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:01.964690 21487 server_base.cc:1034] running on GCE node
W20260620 00:22:01.964428 21492 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:01.965095 21487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:01.965411 21487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:01.966576 21487 hybrid_clock.cc:648] HybridClock initialized: now 1781914921966562 us; error 40 us; skew 500 ppm
Jun 20 00:22:01 dist-test-slave-r81b krb5kdc[21474](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:01.969605 21487 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:22:01.970073 21487 webserver.cc:460] Webserver started at http://127.0.0.1:40401/ using document root <none> and password file <none>
I20260620 00:22:01.970278 21487 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:01.970323 21487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:01.970422 21487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:01.971621 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "7790e438a9584d0797b098d5b5731174"
format_stamp: "Formatted at 2026-06-20 00:22:01 on dist-test-slave-r81b"
server_key: "836dc9674bf8bd26924c085b6f5d7cc2"
server_key_iv: "e271e0e9ecf455f3c4409d51579f532b"
server_key_version: "encryptionkey@0"
I20260620 00:22:01.971964 21487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "7790e438a9584d0797b098d5b5731174"
format_stamp: "Formatted at 2026-06-20 00:22:01 on dist-test-slave-r81b"
server_key: "836dc9674bf8bd26924c085b6f5d7cc2"
server_key_iv: "e271e0e9ecf455f3c4409d51579f532b"
server_key_version: "encryptionkey@0"
I20260620 00:22:01.973598 21487 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:22:01.975061 21502 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:01.975420 21487 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:01.975567 21487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "7790e438a9584d0797b098d5b5731174"
format_stamp: "Formatted at 2026-06-20 00:22:01 on dist-test-slave-r81b"
server_key: "836dc9674bf8bd26924c085b6f5d7cc2"
server_key_iv: "e271e0e9ecf455f3c4409d51579f532b"
server_key_version: "encryptionkey@0"
I20260620 00:22:01.975697 21487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:01.979390 21487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:01.981789 21487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:01.981940 21487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:01.986943 21487 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40847
I20260620 00:22:01.986986 21554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40847 every 8 connection(s)
I20260620 00:22:01.987356 21487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:01.988492 21555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:01.990952 21555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174: Bootstrap starting.
I20260620 00:22:01.991611 21555 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:01.991915 21555 log.cc:824] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:01.992833 21555 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174: No bootstrap required, opened a new log
I20260620 00:22:01.994551 21555 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790e438a9584d0797b098d5b5731174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } }
I20260620 00:22:01.994745 21555 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:01.994797 21555 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7790e438a9584d0797b098d5b5731174, State: Initialized, Role: FOLLOWER
I20260620 00:22:01.994982 21555 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790e438a9584d0797b098d5b5731174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } }
I20260620 00:22:01.995072 21555 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:01.995117 21555 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:01.995167 21555 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:01.995798 21555 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790e438a9584d0797b098d5b5731174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } }
I20260620 00:22:01.995926 21555 leader_election.cc:302] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7790e438a9584d0797b098d5b5731174; no voters:
I20260620 00:22:01.996189 21555 leader_election.cc:288] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:01.996294 21558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:01.996481 21558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [term 1 LEADER]: Becoming Leader. State: Replica: 7790e438a9584d0797b098d5b5731174, State: Running, Role: LEADER
I20260620 00:22:01.996505 21555 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:01.996640 21558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790e438a9584d0797b098d5b5731174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } }
I20260620 00:22:01.997037 21559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7790e438a9584d0797b098d5b5731174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790e438a9584d0797b098d5b5731174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } } }
I20260620 00:22:01.997051 21560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7790e438a9584d0797b098d5b5731174. Latest consensus state: current_term: 1 leader_uuid: "7790e438a9584d0797b098d5b5731174" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7790e438a9584d0797b098d5b5731174" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40847 } } }
I20260620 00:22:01.997190 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 21487
I20260620 00:22:01.997208 21559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:01.997304 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:01.997222 21560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:01.997501 392 external_mini_cluster.cc:1428] Setting key a947e34d61d2970cb8662271457756e8
I20260620 00:22:01.997948 21565 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:01.998538 21565 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:02.000362 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:01.998035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:53641 (local address 127.0.0.1:40847)
0620 00:22:01.998459 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:01.998470 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:01.998734 (+ 264us) server_negotiation.cc:407] Connection header received
0620 00:22:01.998913 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:01.998921 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:01.998954 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:01.999121 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:01.999815 (+ 694us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:01.999819 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:01.999820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:01.999824 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:01.999866 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:01.999881 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:01.999986 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:02.000063 (+ 77us) server_negotiation.cc:299] Negotiation successful
0620 00:22:02.000112 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":76,"threads_started":1}
I20260620 00:22:02.000650 21565 catalog_manager.cc:1347] Generated new cluster ID: 45637196303b482b9e907b8a83c47128
I20260620 00:22:02.000697 21565 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:02.001284 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:02.004119 21573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:02.003115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:49017 (local address 127.0.0.1:40847)
0620 00:22:02.003147 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:02.003149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:02.003157 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:22:02.003222 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:02.003223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:02.003228 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:02.003279 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:02.003905 (+ 626us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:02.003907 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:02.003907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:02.003911 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:02.003927 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:02.003931 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:02.003986 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:02.004039 (+ 53us) server_negotiation.cc:299] Negotiation successful
0620 00:22:02.004053 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260620 00:22:02.009122 21565 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:02.009323 21565 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:02.015201 21565 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7790e438a9584d0797b098d5b5731174: Generated new TSK 0
I20260620 00:22:02.015408 21565 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:02.023147 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 21487
2026-06-20T00:22:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1344 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:02 dist-test-slave-r81b krb5kdc[21586](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:02 dist-test-slave-r81b krb5kdc[21586](info): set up 2 sockets
Jun 20 00:22:02 dist-test-slave-r81b krb5kdc[21586](info): commencing operation
krb5kdc: starting...
W20260620 00:22:03.076453 392 mini_kdc.cc:121] Time spent starting KDC: real 1.034s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:03 dist-test-slave-r81b krb5kdc[21586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:22:03.203104 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43589
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37029
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43589
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:03.285046 21599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:03.285276 21599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:03.285301 21599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:03.286553 21599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:03.286607 21599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:03.286621 21599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:03.286635 21599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:03.286648 21599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:03.288335 21599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37029
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43589
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43589
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:03.288620 21599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:03.288864 21599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:03.293983 21607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:03.293978 21605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:03.293987 21604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:03.294426 21599 server_base.cc:1034] running on GCE node
I20260620 00:22:03.294636 21599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:03.294893 21599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:03.296059 21599 hybrid_clock.cc:648] HybridClock initialized: now 1781914923296042 us; error 36 us; skew 500 ppm
Jun 20 00:22:03 dist-test-slave-r81b krb5kdc[21586](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:03.298703 21599 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:22:03.299175 21599 webserver.cc:460] Webserver started at http://127.0.0.1:33451/ using document root <none> and password file <none>
I20260620 00:22:03.299407 21599 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:03.299477 21599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:03.299635 21599 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:03.300910 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "a63f3289a8b648729b7b5dfb87d3e480"
format_stamp: "Formatted at 2026-06-20 00:22:03 on dist-test-slave-r81b"
server_key: "3144223232bc0e4d3458c0b48342c38e"
server_key_iv: "1679e08ea781850f40a488d972aebc1f"
server_key_version: "encryptionkey@0"
I20260620 00:22:03.301287 21599 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "a63f3289a8b648729b7b5dfb87d3e480"
format_stamp: "Formatted at 2026-06-20 00:22:03 on dist-test-slave-r81b"
server_key: "3144223232bc0e4d3458c0b48342c38e"
server_key_iv: "1679e08ea781850f40a488d972aebc1f"
server_key_version: "encryptionkey@0"
I20260620 00:22:03.302805 21599 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:22:03.303786 21614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:03.304047 21599 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:03.304219 21599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "a63f3289a8b648729b7b5dfb87d3e480"
format_stamp: "Formatted at 2026-06-20 00:22:03 on dist-test-slave-r81b"
server_key: "3144223232bc0e4d3458c0b48342c38e"
server_key_iv: "1679e08ea781850f40a488d972aebc1f"
server_key_version: "encryptionkey@0"
I20260620 00:22:03.304323 21599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:03.316205 21599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:03.318668 21599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:03.318845 21599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:03.322944 21599 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43589
I20260620 00:22:03.322992 21666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43589 every 8 connection(s)
I20260620 00:22:03.323388 21599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:03.324352 21667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:03.326601 21667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480: Bootstrap starting.
I20260620 00:22:03.327133 21667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:03.327363 21667 log.cc:824] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:03.328012 21667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480: No bootstrap required, opened a new log
I20260620 00:22:03.328295 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 21599
I20260620 00:22:03.328421 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:03.328611 392 external_mini_cluster.cc:1428] Setting key 1b6e0818189624671e72ea9ea968e9a4
I20260620 00:22:03.330361 21667 raft_consensus.cc:348] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63f3289a8b648729b7b5dfb87d3e480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } }
I20260620 00:22:03.330601 21667 raft_consensus.cc:374] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:03.330658 21667 raft_consensus.cc:729] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a63f3289a8b648729b7b5dfb87d3e480, State: Initialized, Role: FOLLOWER
I20260620 00:22:03.330803 21667 consensus_queue.cc:260] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63f3289a8b648729b7b5dfb87d3e480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } }
I20260620 00:22:03.330901 21667 raft_consensus.cc:388] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:03.330935 21667 raft_consensus.cc:482] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:03.330982 21667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:03.331566 21667 raft_consensus.cc:504] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63f3289a8b648729b7b5dfb87d3e480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } }
I20260620 00:22:03.331650 21667 leader_election.cc:302] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a63f3289a8b648729b7b5dfb87d3e480; no voters:
I20260620 00:22:03.331831 21667 leader_election.cc:288] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:03.331858 21672 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:03.331967 21672 raft_consensus.cc:686] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [term 1 LEADER]: Becoming Leader. State: Replica: a63f3289a8b648729b7b5dfb87d3e480, State: Running, Role: LEADER
I20260620 00:22:03.332093 21667 sys_catalog.cc:564] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:03.332123 21672 consensus_queue.cc:237] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63f3289a8b648729b7b5dfb87d3e480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } }
I20260620 00:22:03.332513 21672 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a63f3289a8b648729b7b5dfb87d3e480. Latest consensus state: current_term: 1 leader_uuid: "a63f3289a8b648729b7b5dfb87d3e480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63f3289a8b648729b7b5dfb87d3e480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } } }
I20260620 00:22:03.332630 21672 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:03.332583 21673 sys_catalog.cc:455] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a63f3289a8b648729b7b5dfb87d3e480" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a63f3289a8b648729b7b5dfb87d3e480" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43589 } } }
I20260620 00:22:03.332948 21673 sys_catalog.cc:458] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480 [sys.catalog]: This master's current role is: LEADER
W20260620 00:22:03.334030 21688 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:03.334105 21688 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:03.334196 21679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:03.334363 21679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:03.336180 21679 catalog_manager.cc:1347] Generated new cluster ID: 54c2952fa1a945fc850e594ac1b7ccea
I20260620 00:22:03.336242 21679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:22:03 dist-test-slave-r81b krb5kdc[21586](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260620 00:22:03.340570 21671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:03.329133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:49645 (local address 127.0.0.1:43589)
0620 00:22:03.329327 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:03.329335 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:03.329360 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:22:03.329887 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:03.329893 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:03.329942 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:03.330057 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:03.330904 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:03.332542 (+ 1638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:03.333796 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:03.334404 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:03.334541 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:03.337249 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:03.337253 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:03.337253 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:03.337266 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:03.338988 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:03.339406 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:03.339409 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:03.339409 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:03.339499 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:03.339738 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:03.339740 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:03.339740 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:03.339975 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:03.340209 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:03.340309 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:22:03.340410 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260620 00:22:03.341516 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:03.350900 21671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:03.342917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:52451 (local address 127.0.0.1:43589)
0620 00:22:03.342955 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:03.342956 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:03.344341 (+ 1385us) server_negotiation.cc:407] Connection header received
0620 00:22:03.344415 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:03.344416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:03.344426 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:03.344482 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:03.345006 (+ 524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:03.345540 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:03.346286 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:03.346509 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:03.346587 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:03.347827 (+ 1240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:03.347828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:03.347829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:03.347836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:03.349313 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:03.349703 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:03.349705 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:03.349705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:03.349764 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:03.349995 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:03.349995 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:03.349996 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:03.350110 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:03.350233 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:03.350777 (+ 544us) server_negotiation.cc:299] Negotiation successful
0620 00:22:03.350837 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:22:03.350929 21679 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:03.351605 21679 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:03.357144 21679 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a63f3289a8b648729b7b5dfb87d3e480: Generated new TSK 0
I20260620 00:22:03.357353 21679 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:03.363744 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 21599
W20260620 00:22:03.368338 21584 connection.cc:535] client connection to 127.0.0.1:43589 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43589): Connection reset by peer (error 104)
2026-06-20T00:22:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1340 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:03 dist-test-slave-r81b krb5kdc[21698](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:03 dist-test-slave-r81b krb5kdc[21698](info): set up 2 sockets
Jun 20 00:22:03 dist-test-slave-r81b krb5kdc[21698](info): commencing operation
krb5kdc: starting...
W20260620 00:22:04.424499 392 mini_kdc.cc:121] Time spent starting KDC: real 1.042s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:04 dist-test-slave-r81b krb5kdc[21698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914924, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:22:04.547499 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34535
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34535
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:04.629436 21711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:04.629662 21711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:04.629688 21711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:04.630849 21711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:04.630888 21711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:04.630903 21711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:04.630915 21711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:04.630928 21711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:04.632385 21711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34535
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34535
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21711
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:04.632594 21711 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:04.632776 21711 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:04.637889 21719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:04.637931 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:04.637912 21711 server_base.cc:1034] running on GCE node
W20260620 00:22:04.637956 21716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:04.638330 21711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:04.638643 21711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:04.639811 21711 hybrid_clock.cc:648] HybridClock initialized: now 1781914924639797 us; error 53 us; skew 500 ppm
Jun 20 00:22:04 dist-test-slave-r81b krb5kdc[21698](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:04.647677 21711 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:22:04.648090 21711 webserver.cc:460] Webserver started at http://127.0.0.1:40405/ using document root <none> and password file <none>
I20260620 00:22:04.648348 21711 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:04.648397 21711 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:04.648487 21711 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:04.649526 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "9f729893dae14e9e9293e4cf7d68e299"
format_stamp: "Formatted at 2026-06-20 00:22:04 on dist-test-slave-r81b"
server_key: "2d36f101c0b456f54838b34a30ab419b"
server_key_iv: "c06c3a43c90ff5f91b19fb0cae072600"
server_key_version: "encryptionkey@0"
I20260620 00:22:04.649835 21711 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f729893dae14e9e9293e4cf7d68e299"
format_stamp: "Formatted at 2026-06-20 00:22:04 on dist-test-slave-r81b"
server_key: "2d36f101c0b456f54838b34a30ab419b"
server_key_iv: "c06c3a43c90ff5f91b19fb0cae072600"
server_key_version: "encryptionkey@0"
I20260620 00:22:04.651124 21711 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260620 00:22:04.651825 21726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:04.652038 21711 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:04.652177 21711 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "9f729893dae14e9e9293e4cf7d68e299"
format_stamp: "Formatted at 2026-06-20 00:22:04 on dist-test-slave-r81b"
server_key: "2d36f101c0b456f54838b34a30ab419b"
server_key_iv: "c06c3a43c90ff5f91b19fb0cae072600"
server_key_version: "encryptionkey@0"
I20260620 00:22:04.652258 21711 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:04.667142 21711 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:04.669636 21711 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:04.669787 21711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:04.673875 21711 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34535
I20260620 00:22:04.673931 21778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34535 every 8 connection(s)
I20260620 00:22:04.674288 21711 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:04.675340 21779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:04.677793 21779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299: Bootstrap starting.
I20260620 00:22:04.678409 21779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:04.678664 21779 log.cc:824] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:04.679421 21779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299: No bootstrap required, opened a new log
I20260620 00:22:04.681093 21779 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f729893dae14e9e9293e4cf7d68e299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34535 } }
I20260620 00:22:04.681257 21779 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:04.681281 21779 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f729893dae14e9e9293e4cf7d68e299, State: Initialized, Role: FOLLOWER
I20260620 00:22:04.681396 21779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f729893dae14e9e9293e4cf7d68e299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34535 } }
I20260620 00:22:04.681494 21779 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:04.681535 21779 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:04.681624 21779 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:04.682260 21779 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f729893dae14e9e9293e4cf7d68e299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34535 } }
I20260620 00:22:04.682392 21779 leader_election.cc:302] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f729893dae14e9e9293e4cf7d68e299; no voters:
I20260620 00:22:04.682611 21779 leader_election.cc:288] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:04.682659 21782 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:04.682878 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 21711
I20260620 00:22:04.682888 21779 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:04.682928 21782 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [term 1 LEADER]: Becoming Leader. State: Replica: 9f729893dae14e9e9293e4cf7d68e299, State: Running, Role: LEADER
I20260620 00:22:04.682994 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:04.683079 21782 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f729893dae14e9e9293e4cf7d68e299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34535 } }
I20260620 00:22:04.683212 392 external_mini_cluster.cc:1428] Setting key 071cdb2bea9e7cdf621299601a816bb1
I20260620 00:22:04.683913 21784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f729893dae14e9e9293e4cf7d68e299. Latest consensus state: current_term: 1 leader_uuid: "9f729893dae14e9e9293e4cf7d68e299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f729893dae14e9e9293e4cf7d68e299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34535 } } }
I20260620 00:22:04.684053 21783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f729893dae14e9e9293e4cf7d68e299" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f729893dae14e9e9293e4cf7d68e299" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34535 } } }
I20260620 00:22:04.684054 21784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:04.684120 21783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:04.684413 21798 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:04.685072 21798 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:04.686556 21798 catalog_manager.cc:1347] Generated new cluster ID: e519fedc14b84ebf90cbb89a921611d1
I20260620 00:22:04.686609 21798 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:04.690095 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:04.683756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:49411 (local address 127.0.0.1:34535)
0620 00:22:04.684564 (+ 808us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:04.684574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:04.684611 (+ 37us) server_negotiation.cc:407] Connection header received
0620 00:22:04.684751 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:04.684760 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:04.684825 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:04.684989 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:04.685253 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:04.686623 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:04.687756 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:04.688483 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:04.688610 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:04.689368 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:04.689373 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:04.689373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:04.689599 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:04.689641 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:04.689658 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:04.689743 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:04.689835 (+ 92us) server_negotiation.cc:299] Negotiation successful
0620 00:22:04.689866 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":77,"threads_started":1}
I20260620 00:22:04.691123 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:04.699345 21792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:04.692426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:60311 (local address 127.0.0.1:34535)
0620 00:22:04.692480 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:04.692482 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:04.696399 (+ 3917us) server_negotiation.cc:407] Connection header received
0620 00:22:04.696486 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:04.696486 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:04.696500 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:04.696560 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:04.696731 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:04.697232 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:04.697986 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:04.698217 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:04.698309 (+ 92us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:04.699060 (+ 751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:04.699061 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:04.699061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:04.699067 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:04.699083 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:04.699088 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:04.699147 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:04.699220 (+ 73us) server_negotiation.cc:299] Negotiation successful
0620 00:22:04.699235 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:22:04.700661 21798 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:04.701197 21798 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:04.707777 21798 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f729893dae14e9e9293e4cf7d68e299: Generated new TSK 0
I20260620 00:22:04.708021 21798 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:04.712707 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 21711
W20260620 00:22:04.716900 21696 connection.cc:535] client connection to 127.0.0.1:34535 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34535): Connection reset by peer (error 104)
2026-06-20T00:22:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1348 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:04 dist-test-slave-r81b krb5kdc[21810](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:04 dist-test-slave-r81b krb5kdc[21810](info): set up 2 sockets
Jun 20 00:22:04 dist-test-slave-r81b krb5kdc[21810](info): commencing operation
krb5kdc: starting...
W20260620 00:22:05.768371 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:05 dist-test-slave-r81b krb5kdc[21810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914925, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260620 00:22:05.895759 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34283
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34283
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:05.988885 21823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:05.989184 21823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:05.989229 21823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:05.990511 21823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:05.990594 21823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:05.990619 21823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:05.990640 21823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:05.990667 21823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:05.992272 21823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34283
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34283
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:05.992563 21823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:05.992799 21823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:05.997685 21828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:05.997764 21831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:05.997781 21829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:05.997855 21823 server_base.cc:1034] running on GCE node
I20260620 00:22:05.998183 21823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:05.998427 21823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:05.999579 21823 hybrid_clock.cc:648] HybridClock initialized: now 1781914925999571 us; error 37 us; skew 500 ppm
Jun 20 00:22:06 dist-test-slave-r81b krb5kdc[21810](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914926, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:06.002331 21823 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260620 00:22:06.002892 21823 webserver.cc:460] Webserver started at http://127.0.0.1:37707/ using document root <none> and password file <none>
I20260620 00:22:06.003147 21823 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:06.003219 21823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:06.003368 21823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:06.004563 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "e9b1d9c3d6104d0daa962344014964e2"
format_stamp: "Formatted at 2026-06-20 00:22:06 on dist-test-slave-r81b"
server_key: "e7dec476c588f76295d1b4d9a7eacab6"
server_key_iv: "0e57d849536f5ca2e9267b86d9ad42d8"
server_key_version: "encryptionkey@0"
I20260620 00:22:06.004925 21823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "e9b1d9c3d6104d0daa962344014964e2"
format_stamp: "Formatted at 2026-06-20 00:22:06 on dist-test-slave-r81b"
server_key: "e7dec476c588f76295d1b4d9a7eacab6"
server_key_iv: "0e57d849536f5ca2e9267b86d9ad42d8"
server_key_version: "encryptionkey@0"
I20260620 00:22:06.006350 21823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:22:06.007215 21838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:06.007444 21823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:06.007555 21823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "e9b1d9c3d6104d0daa962344014964e2"
format_stamp: "Formatted at 2026-06-20 00:22:06 on dist-test-slave-r81b"
server_key: "e7dec476c588f76295d1b4d9a7eacab6"
server_key_iv: "0e57d849536f5ca2e9267b86d9ad42d8"
server_key_version: "encryptionkey@0"
I20260620 00:22:06.007653 21823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:06.011171 21823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:06.013628 21823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:06.013895 21823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:06.018121 21823 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34283
I20260620 00:22:06.018151 21890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34283 every 8 connection(s)
I20260620 00:22:06.018540 21823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:06.019681 21891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:06.021657 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 21823
I20260620 00:22:06.021800 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:06.021996 392 external_mini_cluster.cc:1428] Setting key cdf4ee5cefa2dd48bffb9ef38dc0e09c
I20260620 00:22:06.022783 21891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2: Bootstrap starting.
I20260620 00:22:06.023444 21891 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:06.023828 21891 log.cc:824] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:06.024708 21891 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2: No bootstrap required, opened a new log
I20260620 00:22:06.024870 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:06.022601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:34217 (local address 127.0.0.1:34283)
0620 00:22:06.022812 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:06.022820 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:06.022845 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:22:06.023335 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:06.023342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:06.023370 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:06.023578 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:06.024322 (+ 744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:06.024326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:06.024326 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:06.024330 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:06.024398 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:06.024413 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:06.024583 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:06.024620 (+ 37us) server_negotiation.cc:299] Negotiation successful
0620 00:22:06.024665 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":94,"threads_started":1}
I20260620 00:22:06.025754 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:06.027014 21891 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b1d9c3d6104d0daa962344014964e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } }
I20260620 00:22:06.027328 21891 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:06.027398 21891 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9b1d9c3d6104d0daa962344014964e2, State: Initialized, Role: FOLLOWER
I20260620 00:22:06.027534 21891 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b1d9c3d6104d0daa962344014964e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } }
I20260620 00:22:06.027652 21891 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:06.027688 21891 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:06.027740 21891 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:06.028463 21894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:06.027203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.226:38399 (local address 127.0.0.1:34283)
0620 00:22:06.027245 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:06.027246 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:06.027437 (+ 191us) server_negotiation.cc:407] Connection header received
0620 00:22:06.027530 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:06.027532 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:06.027537 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:06.027612 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:06.028244 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:06.028246 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:06.028246 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:06.028250 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:06.028265 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:06.028269 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:06.028337 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:06.028371 (+ 34us) server_negotiation.cc:299] Negotiation successful
0620 00:22:06.028385 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260620 00:22:06.028482 21891 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b1d9c3d6104d0daa962344014964e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } }
I20260620 00:22:06.028606 21891 leader_election.cc:302] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9b1d9c3d6104d0daa962344014964e2; no voters:
I20260620 00:22:06.028769 21891 leader_election.cc:288] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:06.028816 21901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:06.028990 21901 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [term 1 LEADER]: Becoming Leader. State: Replica: e9b1d9c3d6104d0daa962344014964e2, State: Running, Role: LEADER
I20260620 00:22:06.029058 21891 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:06.029088 21901 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b1d9c3d6104d0daa962344014964e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } }
I20260620 00:22:06.029420 21901 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9b1d9c3d6104d0daa962344014964e2. Latest consensus state: current_term: 1 leader_uuid: "e9b1d9c3d6104d0daa962344014964e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b1d9c3d6104d0daa962344014964e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } } }
I20260620 00:22:06.029435 21903 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9b1d9c3d6104d0daa962344014964e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9b1d9c3d6104d0daa962344014964e2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34283 } } }
I20260620 00:22:06.029580 21901 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:06.029676 21903 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:06.029888 21910 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:06.030478 21910 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:06.032111 21910 catalog_manager.cc:1347] Generated new cluster ID: 7be123ae97264145a12b58477fd17ada
I20260620 00:22:06.032193 21910 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:06.043004 21910 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:06.043203 21910 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:06.058550 21910 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9b1d9c3d6104d0daa962344014964e2: Generated new TSK 0
I20260620 00:22:06.058797 21910 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:06.065337 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 21823
2026-06-20T00:22:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1353 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:06 dist-test-slave-r81b krb5kdc[21922](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:06 dist-test-slave-r81b krb5kdc[21922](info): set up 2 sockets
Jun 20 00:22:06 dist-test-slave-r81b krb5kdc[21922](info): commencing operation
krb5kdc: starting...
W20260620 00:22:07.124488 392 mini_kdc.cc:121] Time spent starting KDC: real 1.040s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:07 dist-test-slave-r81b krb5kdc[21922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:22:07.253386 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44041
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37949
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:07.337226 21935 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:07.337494 21935 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:07.337535 21935 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:07.338774 21935 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:07.338848 21935 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:07.338873 21935 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:07.338896 21935 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:07.338923 21935 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:07.340474 21935 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37949
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21935
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:07.340773 21935 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:07.341013 21935 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:07.346077 21943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:07.346091 21941 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:07.346125 21940 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:07.346228 21935 server_base.cc:1034] running on GCE node
I20260620 00:22:07.346588 21935 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:07.346890 21935 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:07.348112 21935 hybrid_clock.cc:648] HybridClock initialized: now 1781914927348062 us; error 61 us; skew 500 ppm
Jun 20 00:22:07 dist-test-slave-r81b krb5kdc[21922](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914927, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:07.351207 21935 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:22:07.351635 21935 webserver.cc:460] Webserver started at http://127.0.98.62:35361/ using document root <none> and password file <none>
I20260620 00:22:07.351851 21935 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:07.351893 21935 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:07.351998 21935 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:07.353199 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "b55536ba08ab451d9f2cfa173a4a30df"
format_stamp: "Formatted at 2026-06-20 00:22:07 on dist-test-slave-r81b"
server_key: "262ed5d60b9bb753b83cfbd1ae6678fa"
server_key_iv: "d61a52783238a65f4cea018a506cc03f"
server_key_version: "encryptionkey@0"
I20260620 00:22:07.353506 21935 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "b55536ba08ab451d9f2cfa173a4a30df"
format_stamp: "Formatted at 2026-06-20 00:22:07 on dist-test-slave-r81b"
server_key: "262ed5d60b9bb753b83cfbd1ae6678fa"
server_key_iv: "d61a52783238a65f4cea018a506cc03f"
server_key_version: "encryptionkey@0"
I20260620 00:22:07.354809 21935 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:22:07.355544 21950 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:07.355752 21935 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:07.355845 21935 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "b55536ba08ab451d9f2cfa173a4a30df"
format_stamp: "Formatted at 2026-06-20 00:22:07 on dist-test-slave-r81b"
server_key: "262ed5d60b9bb753b83cfbd1ae6678fa"
server_key_iv: "d61a52783238a65f4cea018a506cc03f"
server_key_version: "encryptionkey@0"
I20260620 00:22:07.355908 21935 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:07.376230 21935 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:07.378669 21935 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:07.378786 21935 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:07.382653 21935 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44041
I20260620 00:22:07.382717 22002 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44041 every 8 connection(s)
I20260620 00:22:07.383106 21935 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:07.384210 22003 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:07.386523 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df: Bootstrap starting.
I20260620 00:22:07.387131 22003 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:07.387451 22003 log.cc:824] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:07.388268 22003 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df: No bootstrap required, opened a new log
I20260620 00:22:07.388600 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 21935
I20260620 00:22:07.388728 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:07.388914 392 external_mini_cluster.cc:1428] Setting key 0c04fffc21b19d799216d1fb844c52d0
I20260620 00:22:07.390633 22003 raft_consensus.cc:348] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55536ba08ab451d9f2cfa173a4a30df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44041 } }
I20260620 00:22:07.390808 22003 raft_consensus.cc:374] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:07.390848 22003 raft_consensus.cc:729] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b55536ba08ab451d9f2cfa173a4a30df, State: Initialized, Role: FOLLOWER
I20260620 00:22:07.391039 22003 consensus_queue.cc:260] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55536ba08ab451d9f2cfa173a4a30df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44041 } }
I20260620 00:22:07.391136 22003 raft_consensus.cc:388] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:07.391170 22003 raft_consensus.cc:482] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:07.391217 22003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:07.391844 22003 raft_consensus.cc:504] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55536ba08ab451d9f2cfa173a4a30df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44041 } }
I20260620 00:22:07.391960 22003 leader_election.cc:302] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b55536ba08ab451d9f2cfa173a4a30df; no voters:
I20260620 00:22:07.392154 22003 leader_election.cc:288] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:07.392231 22008 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:07.392410 22003 sys_catalog.cc:564] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:07.392436 22008 raft_consensus.cc:686] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [term 1 LEADER]: Becoming Leader. State: Replica: b55536ba08ab451d9f2cfa173a4a30df, State: Running, Role: LEADER
I20260620 00:22:07.392607 22008 consensus_queue.cc:237] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55536ba08ab451d9f2cfa173a4a30df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44041 } }
I20260620 00:22:07.394076 22009 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b55536ba08ab451d9f2cfa173a4a30df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55536ba08ab451d9f2cfa173a4a30df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44041 } } }
I20260620 00:22:07.394222 22010 sys_catalog.cc:455] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [sys.catalog]: SysCatalogTable state changed. Reason: New leader b55536ba08ab451d9f2cfa173a4a30df. Latest consensus state: current_term: 1 leader_uuid: "b55536ba08ab451d9f2cfa173a4a30df" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b55536ba08ab451d9f2cfa173a4a30df" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44041 } } }
I20260620 00:22:07.394549 22009 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:07.394560 22010 sys_catalog.cc:458] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df [sys.catalog]: This master's current role is: LEADER
W20260620 00:22:07.395059 22022 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:07.395151 22022 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:07.395244 22024 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:07.395475 22024 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:07.397239 22024 catalog_manager.cc:1347] Generated new cluster ID: 930f9b08143e43afa3fdb0f34ce320b3
I20260620 00:22:07.397297 22024 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Jun 20 00:22:07 dist-test-slave-r81b krb5kdc[21922](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:22:07.403115 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:07.389490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49991 (local address 127.0.98.62:44041)
0620 00:22:07.389723 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:07.389733 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:07.389766 (+ 33us) server_negotiation.cc:407] Connection header received
0620 00:22:07.390580 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:07.390591 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:07.390654 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:07.390807 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:07.391697 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:07.393136 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:07.394398 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:07.395016 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:07.395149 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:07.398350 (+ 3201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:07.398354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:07.398355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:07.398368 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:07.401068 (+ 2700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:07.401537 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:07.401540 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:07.401541 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:07.401643 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:07.401854 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:07.401855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:07.401855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:07.402168 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:07.402322 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:07.402769 (+ 447us) server_negotiation.cc:299] Negotiation successful
0620 00:22:07.402872 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":104,"threads_started":1}
I20260620 00:22:07.404258 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:07.412128 22024 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:07.412863 22024 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:07.416532 22007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:07.408494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36719 (local address 127.0.98.62:44041)
0620 00:22:07.408538 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:07.408540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:07.408656 (+ 116us) server_negotiation.cc:407] Connection header received
0620 00:22:07.408707 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:07.408708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:07.408718 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:07.408788 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:07.409494 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:07.410371 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:07.411458 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:07.411865 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:07.412002 (+ 137us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:07.413562 (+ 1560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:07.413565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:07.413565 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:07.413574 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:07.415165 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:07.415590 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:07.415591 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:07.415591 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:07.415658 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:07.415920 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:07.415921 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:07.415921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:07.416042 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:07.416227 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:07.416351 (+ 124us) server_negotiation.cc:299] Negotiation successful
0620 00:22:07.416418 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":21}
I20260620 00:22:07.420806 22024 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b55536ba08ab451d9f2cfa173a4a30df: Generated new TSK 0
I20260620 00:22:07.421036 22024 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:07.426124 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 21935
W20260620 00:22:07.430773 21920 connection.cc:535] client connection to 127.0.98.62:44041 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:44041): Connection reset by peer (error 104)
2026-06-20T00:22:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:07 dist-test-slave-r81b krb5kdc[22034](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:07 dist-test-slave-r81b krb5kdc[22034](info): set up 2 sockets
Jun 20 00:22:07 dist-test-slave-r81b krb5kdc[22034](info): commencing operation
krb5kdc: starting...
W20260620 00:22:08.520571 392 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:08 dist-test-slave-r81b krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:22:08.643306 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35147
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45959
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35147
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:08.726044 22047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:08.726270 22047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:08.726294 22047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:08.727430 22047 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:08.727473 22047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:08.727487 22047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:08.727500 22047 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:08.727519 22047 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:08.729158 22047 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45959
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35147
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:08.729364 22047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:08.729549 22047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260620 00:22:08.734712 22047 server_base.cc:1034] running on GCE node
W20260620 00:22:08.734663 22052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:08.734669 22055 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:08.734669 22053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:08.735181 22047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:08.735505 22047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:08.736689 22047 hybrid_clock.cc:648] HybridClock initialized: now 1781914928736670 us; error 36 us; skew 500 ppm
Jun 20 00:22:08 dist-test-slave-r81b krb5kdc[22034](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914928, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:08.739331 22047 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:22:08.739873 22047 webserver.cc:460] Webserver started at http://127.0.98.62:41385/ using document root <none> and password file <none>
I20260620 00:22:08.740120 22047 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:08.740244 22047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:08.740404 22047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:08.741539 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "6585291d3c7a428f92cb72f3ad53fc2d"
format_stamp: "Formatted at 2026-06-20 00:22:08 on dist-test-slave-r81b"
server_key: "b3aedd9f794777a900332fe970e4dd3e"
server_key_iv: "f8a08b7f6338dcb0f98508bf2d9a2179"
server_key_version: "encryptionkey@0"
I20260620 00:22:08.741937 22047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "6585291d3c7a428f92cb72f3ad53fc2d"
format_stamp: "Formatted at 2026-06-20 00:22:08 on dist-test-slave-r81b"
server_key: "b3aedd9f794777a900332fe970e4dd3e"
server_key_iv: "f8a08b7f6338dcb0f98508bf2d9a2179"
server_key_version: "encryptionkey@0"
I20260620 00:22:08.743331 22047 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:22:08.744235 22062 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:08.744411 22047 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:08.744511 22047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "6585291d3c7a428f92cb72f3ad53fc2d"
format_stamp: "Formatted at 2026-06-20 00:22:08 on dist-test-slave-r81b"
server_key: "b3aedd9f794777a900332fe970e4dd3e"
server_key_iv: "f8a08b7f6338dcb0f98508bf2d9a2179"
server_key_version: "encryptionkey@0"
I20260620 00:22:08.744608 22047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:08.765237 22047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:08.767498 22047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:08.767686 22047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:08.772027 22047 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35147
I20260620 00:22:08.772074 22114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35147 every 8 connection(s)
I20260620 00:22:08.772493 22047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:08.773372 22115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:08.775614 22115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d: Bootstrap starting.
I20260620 00:22:08.776166 22115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:08.776399 22115 log.cc:824] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:08.777051 22115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d: No bootstrap required, opened a new log
I20260620 00:22:08.778597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 22047
I20260620 00:22:08.778748 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:08.778596 22115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585291d3c7a428f92cb72f3ad53fc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35147 } }
I20260620 00:22:08.778808 22115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:08.778856 22115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6585291d3c7a428f92cb72f3ad53fc2d, State: Initialized, Role: FOLLOWER
I20260620 00:22:08.778959 392 external_mini_cluster.cc:1428] Setting key 9984f7b5536d5d832a1905c35acef714
I20260620 00:22:08.779037 22115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585291d3c7a428f92cb72f3ad53fc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35147 } }
I20260620 00:22:08.779130 22115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:08.779175 22115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:08.779225 22115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:08.780387 22115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585291d3c7a428f92cb72f3ad53fc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35147 } }
I20260620 00:22:08.780510 22115 leader_election.cc:302] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6585291d3c7a428f92cb72f3ad53fc2d; no voters:
I20260620 00:22:08.780678 22115 leader_election.cc:288] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:08.780730 22120 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:08.780876 22120 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [term 1 LEADER]: Becoming Leader. State: Replica: 6585291d3c7a428f92cb72f3ad53fc2d, State: Running, Role: LEADER
I20260620 00:22:08.780958 22115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:08.781009 22120 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585291d3c7a428f92cb72f3ad53fc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35147 } }
I20260620 00:22:08.781420 22120 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6585291d3c7a428f92cb72f3ad53fc2d. Latest consensus state: current_term: 1 leader_uuid: "6585291d3c7a428f92cb72f3ad53fc2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585291d3c7a428f92cb72f3ad53fc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35147 } } }
I20260620 00:22:08.781507 22121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6585291d3c7a428f92cb72f3ad53fc2d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6585291d3c7a428f92cb72f3ad53fc2d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35147 } } }
I20260620 00:22:08.781575 22120 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:08.781613 22121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:08.781792 22129 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:08.782536 22129 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:08.784509 22129 catalog_manager.cc:1347] Generated new cluster ID: 5f4f4ae35faa4a3fad618398fd556ab3
I20260620 00:22:08.784574 22129 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:08.785022 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:08.779497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43165 (local address 127.0.98.62:35147)
0620 00:22:08.779734 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:08.779744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:08.779780 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:22:08.779883 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:08.779889 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:08.779933 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:08.780045 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:08.780321 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:08.781587 (+ 1266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:08.782806 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:08.783305 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:08.783429 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:08.784221 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:08.784225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:08.784226 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:08.784434 (+ 208us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:08.784474 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:08.784489 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:08.784588 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:08.784785 (+ 197us) server_negotiation.cc:299] Negotiation successful
0620 00:22:08.784813 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":148,"threads_started":1}
I20260620 00:22:08.785943 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:08.791319 22119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:08.787311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59715 (local address 127.0.98.62:35147)
0620 00:22:08.787348 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:08.787350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:08.787417 (+ 67us) server_negotiation.cc:407] Connection header received
0620 00:22:08.787527 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:08.787529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:08.787539 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:08.787570 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:08.787758 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:08.788566 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:08.789699 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:08.790062 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:08.790110 (+ 48us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:08.790997 (+ 887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:08.790998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:08.790998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:08.791004 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:08.791020 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:08.791026 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:08.791059 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:08.791200 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 00:22:08.791216 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:22:08.805189 22129 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:08.805996 22129 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:08.813836 22129 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6585291d3c7a428f92cb72f3ad53fc2d: Generated new TSK 0
I20260620 00:22:08.814065 22129 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:08.820104 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 22047
W20260620 00:22:08.824014 22032 connection.cc:535] client connection to 127.0.98.62:35147 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:35147): Connection reset by peer (error 104)
2026-06-20T00:22:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1393 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:08 dist-test-slave-r81b krb5kdc[22146](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:08 dist-test-slave-r81b krb5kdc[22146](info): set up 2 sockets
Jun 20 00:22:08 dist-test-slave-r81b krb5kdc[22146](info): commencing operation
krb5kdc: starting...
W20260620 00:22:09.864378 392 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:09 dist-test-slave-r81b krb5kdc[22146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914929, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:22:09.987217 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37209
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37741
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37209
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:10.070053 22159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:10.070282 22159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:10.070307 22159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:10.071552 22159 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:10.071604 22159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:10.071619 22159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:10.071632 22159 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:10.071645 22159 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:10.073215 22159 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37741
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37209
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:10.073482 22159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:10.073710 22159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:10.078804 22167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:10.078804 22165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:10.078816 22164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:10.079214 22159 server_base.cc:1034] running on GCE node
I20260620 00:22:10.079460 22159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:10.079802 22159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:10.080966 22159 hybrid_clock.cc:648] HybridClock initialized: now 1781914930080955 us; error 30 us; skew 500 ppm
Jun 20 00:22:10 dist-test-slave-r81b krb5kdc[22146](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:10.088825 22159 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:22:10.089408 22159 webserver.cc:460] Webserver started at http://127.0.98.62:41937/ using document root <none> and password file <none>
I20260620 00:22:10.089670 22159 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:10.089721 22159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:10.089862 22159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:10.090989 22159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "9d1819a1f6104190b570142b4c3c7c65"
format_stamp: "Formatted at 2026-06-20 00:22:10 on dist-test-slave-r81b"
server_key: "c4cc8a558d7aa1238235cd1b76f268cc"
server_key_iv: "79e4e193e883fa79cdf2a8f70e1d8ccc"
server_key_version: "encryptionkey@0"
I20260620 00:22:10.091495 22159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "9d1819a1f6104190b570142b4c3c7c65"
format_stamp: "Formatted at 2026-06-20 00:22:10 on dist-test-slave-r81b"
server_key: "c4cc8a558d7aa1238235cd1b76f268cc"
server_key_iv: "79e4e193e883fa79cdf2a8f70e1d8ccc"
server_key_version: "encryptionkey@0"
I20260620 00:22:10.093273 22159 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.004s
I20260620 00:22:10.094244 22174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:10.094470 22159 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:22:10.094558 22159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "9d1819a1f6104190b570142b4c3c7c65"
format_stamp: "Formatted at 2026-06-20 00:22:10 on dist-test-slave-r81b"
server_key: "c4cc8a558d7aa1238235cd1b76f268cc"
server_key_iv: "79e4e193e883fa79cdf2a8f70e1d8ccc"
server_key_version: "encryptionkey@0"
I20260620 00:22:10.094655 22159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:10.098086 22159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:10.100433 22159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:10.100625 22159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:10.104470 22159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37209
I20260620 00:22:10.104501 22226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37209 every 8 connection(s)
I20260620 00:22:10.104848 22159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:10.105878 22227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:10.108238 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65: Bootstrap starting.
I20260620 00:22:10.108799 22227 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:10.109047 22227 log.cc:824] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:10.109925 22227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65: No bootstrap required, opened a new log
I20260620 00:22:10.111740 22227 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1819a1f6104190b570142b4c3c7c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37209 } }
I20260620 00:22:10.111905 22227 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:10.111930 22227 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d1819a1f6104190b570142b4c3c7c65, State: Initialized, Role: FOLLOWER
I20260620 00:22:10.112128 22227 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1819a1f6104190b570142b4c3c7c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37209 } }
I20260620 00:22:10.112267 22227 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:10.112293 22227 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:10.112327 22227 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:10.112708 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 22159
I20260620 00:22:10.112845 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:10.113044 392 external_mini_cluster.cc:1428] Setting key eee6a07fa7508b09a81fe7315cd842e6
I20260620 00:22:10.113214 22227 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1819a1f6104190b570142b4c3c7c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37209 } }
I20260620 00:22:10.113379 22227 leader_election.cc:302] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9d1819a1f6104190b570142b4c3c7c65; no voters:
I20260620 00:22:10.113611 22227 leader_election.cc:288] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:10.113672 22230 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:10.113857 22230 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [term 1 LEADER]: Becoming Leader. State: Replica: 9d1819a1f6104190b570142b4c3c7c65, State: Running, Role: LEADER
I20260620 00:22:10.113992 22227 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:10.114001 22230 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1819a1f6104190b570142b4c3c7c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37209 } }
I20260620 00:22:10.114718 22233 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9d1819a1f6104190b570142b4c3c7c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1819a1f6104190b570142b4c3c7c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37209 } } }
I20260620 00:22:10.114851 22233 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:10.115899 22234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d1819a1f6104190b570142b4c3c7c65. Latest consensus state: current_term: 1 leader_uuid: "9d1819a1f6104190b570142b4c3c7c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9d1819a1f6104190b570142b4c3c7c65" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37209 } } }
I20260620 00:22:10.116088 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:10.113587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33921 (local address 127.0.98.62:37209)
0620 00:22:10.113814 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:10.113826 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:10.114026 (+ 200us) server_negotiation.cc:407] Connection header received
0620 00:22:10.114199 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:10.114208 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:10.114242 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:10.114395 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:10.115293 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:10.115298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:10.115298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:10.115305 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:10.115364 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:10.115381 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:10.115428 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:10.115702 (+ 274us) server_negotiation.cc:299] Negotiation successful
0620 00:22:10.115761 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":77,"threads_started":1}
I20260620 00:22:10.116169 22234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65 [sys.catalog]: This master's current role is: LEADER
W20260620 00:22:10.116631 22248 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:10.116698 22248 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:10.116751 22238 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:10.116946 22238 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:10.117477 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:10.118731 22238 catalog_manager.cc:1347] Generated new cluster ID: 82e645e9286e44e3ba242ddef473e98c
I20260620 00:22:10.118801 22238 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:10.120565 22232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:10.119190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45999 (local address 127.0.98.62:37209)
0620 00:22:10.119229 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:10.119231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:10.119418 (+ 187us) server_negotiation.cc:407] Connection header received
0620 00:22:10.119447 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:10.119448 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:10.119454 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:10.119516 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:10.120225 (+ 709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:10.120226 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:10.120227 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:10.120231 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:10.120249 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:10.120254 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:10.120321 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:10.120393 (+ 72us) server_negotiation.cc:299] Negotiation successful
0620 00:22:10.120405 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:22:10.135260 22238 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:10.135509 22238 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:10.141453 22238 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9d1819a1f6104190b570142b4c3c7c65: Generated new TSK 0
I20260620 00:22:10.141705 22238 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:10.145406 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 22159
2026-06-20T00:22:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:10 dist-test-slave-r81b krb5kdc[22258](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:10 dist-test-slave-r81b krb5kdc[22258](info): set up 2 sockets
Jun 20 00:22:10 dist-test-slave-r81b krb5kdc[22258](info): commencing operation
krb5kdc: starting...
W20260620 00:22:11.212468 392 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:11 dist-test-slave-r81b krb5kdc[22258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:22:11.336280 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41955
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:11.419139 22271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:11.419396 22271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:11.419436 22271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:11.420692 22271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:11.420764 22271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:11.420790 22271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:11.420836 22271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:11.420864 22271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:11.422372 22271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:11.422664 22271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:11.422901 22271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:11.428026 22279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:11.428054 22276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:11.427995 22277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:11.428328 22271 server_base.cc:1034] running on GCE node
I20260620 00:22:11.428568 22271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:11.428876 22271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:11.430055 22271 hybrid_clock.cc:648] HybridClock initialized: now 1781914931430051 us; error 55 us; skew 500 ppm
Jun 20 00:22:11 dist-test-slave-r81b krb5kdc[22258](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:11.432904 22271 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:22:11.433411 22271 webserver.cc:460] Webserver started at http://127.0.98.62:44101/ using document root <none> and password file <none>
I20260620 00:22:11.433678 22271 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:11.433750 22271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:11.433909 22271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:11.435019 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "077570db550b4a8a9cc105341689b46e"
format_stamp: "Formatted at 2026-06-20 00:22:11 on dist-test-slave-r81b"
server_key: "113ca0df4d027a87d2ea10dbaaf83bf5"
server_key_iv: "1751ec505d2d2b000ba4e74ac021013e"
server_key_version: "encryptionkey@0"
I20260620 00:22:11.435400 22271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "077570db550b4a8a9cc105341689b46e"
format_stamp: "Formatted at 2026-06-20 00:22:11 on dist-test-slave-r81b"
server_key: "113ca0df4d027a87d2ea10dbaaf83bf5"
server_key_iv: "1751ec505d2d2b000ba4e74ac021013e"
server_key_version: "encryptionkey@0"
I20260620 00:22:11.436903 22271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:22:11.437705 22286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:11.438084 22271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:22:11.438197 22271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "077570db550b4a8a9cc105341689b46e"
format_stamp: "Formatted at 2026-06-20 00:22:11 on dist-test-slave-r81b"
server_key: "113ca0df4d027a87d2ea10dbaaf83bf5"
server_key_iv: "1751ec505d2d2b000ba4e74ac021013e"
server_key_version: "encryptionkey@0"
I20260620 00:22:11.438274 22271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:11.460777 22271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:11.463279 22271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:11.463529 22271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:11.467634 22271 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41955
I20260620 00:22:11.467669 22338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41955 every 8 connection(s)
I20260620 00:22:11.468025 22271 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:11.469017 22339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:11.471392 22339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e: Bootstrap starting.
I20260620 00:22:11.471518 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 22271
I20260620 00:22:11.471638 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:11.471833 392 external_mini_cluster.cc:1428] Setting key 3b168af5672850adf8c03af180d211df
I20260620 00:22:11.472061 22339 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:11.472476 22339 log.cc:824] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:11.473292 22339 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e: No bootstrap required, opened a new log
I20260620 00:22:11.475534 22339 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077570db550b4a8a9cc105341689b46e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } }
I20260620 00:22:11.475728 22339 raft_consensus.cc:374] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:11.475764 22339 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077570db550b4a8a9cc105341689b46e, State: Initialized, Role: FOLLOWER
I20260620 00:22:11.475889 22339 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077570db550b4a8a9cc105341689b46e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } }
I20260620 00:22:11.475960 22339 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:11.475989 22339 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:11.476027 22339 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:11.476724 22339 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077570db550b4a8a9cc105341689b46e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } }
I20260620 00:22:11.476858 22339 leader_election.cc:302] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077570db550b4a8a9cc105341689b46e; no voters:
I20260620 00:22:11.477046 22339 leader_election.cc:288] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:11.477072 22344 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:11.477183 22344 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [term 1 LEADER]: Becoming Leader. State: Replica: 077570db550b4a8a9cc105341689b46e, State: Running, Role: LEADER
I20260620 00:22:11.477324 22344 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077570db550b4a8a9cc105341689b46e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } }
I20260620 00:22:11.477325 22339 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:11.477777 22345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "077570db550b4a8a9cc105341689b46e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077570db550b4a8a9cc105341689b46e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } } }
I20260620 00:22:11.477788 22346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077570db550b4a8a9cc105341689b46e. Latest consensus state: current_term: 1 leader_uuid: "077570db550b4a8a9cc105341689b46e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077570db550b4a8a9cc105341689b46e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41955 } } }
I20260620 00:22:11.477877 22345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:11.477880 22346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e [sys.catalog]: This master's current role is: LEADER
W20260620 00:22:11.479489 22360 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:11.479569 22360 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Jun 20 00:22:11 dist-test-slave-r81b krb5kdc[22258](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914931, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:22:11.479657 22348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:11.479869 22348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:11.482089 22348 catalog_manager.cc:1347] Generated new cluster ID: 7c33dee716c448dea2e4b96f092fbc45
I20260620 00:22:11.482147 22348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:11.484440 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:11.472414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44315 (local address 127.0.98.62:41955)
0620 00:22:11.472598 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:11.472614 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:11.472639 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:22:11.473066 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:11.473072 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:11.473119 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:11.473240 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:11.474070 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:11.475189 (+ 1119us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:11.476286 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:11.476793 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:11.476950 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:11.480260 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:11.480265 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:11.480265 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:11.480280 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:11.482523 (+ 2243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:11.482995 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:11.482999 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:11.482999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:11.483103 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:11.483343 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:11.483344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:11.483345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:11.483620 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:11.483795 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:11.483934 (+ 139us) server_negotiation.cc:299] Negotiation successful
0620 00:22:11.484253 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":94,"threads_started":1}
I20260620 00:22:11.485421 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:11.492782 22348 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:11.493579 22348 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:11.499243 22348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077570db550b4a8a9cc105341689b46e: Generated new TSK 0
I20260620 00:22:11.499460 22348 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:11.500452 22342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:11.487053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56815 (local address 127.0.98.62:41955)
0620 00:22:11.487097 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:11.487099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:11.492601 (+ 5502us) server_negotiation.cc:407] Connection header received
0620 00:22:11.492687 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:11.492689 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:11.492701 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:11.492772 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:11.493599 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:11.494369 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:11.495421 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:11.495782 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:11.495857 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:11.497376 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:11.497377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:11.497378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:11.497386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:11.499159 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:11.499578 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:11.499579 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:11.499579 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:11.499636 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:11.499855 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:11.499856 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:11.499856 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:11.500004 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:11.500106 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:11.500264 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 00:22:11.500358 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:22:11.502559 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 22271
2026-06-20T00:22:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1356 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:11 dist-test-slave-r81b krb5kdc[22370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:11 dist-test-slave-r81b krb5kdc[22370](info): set up 2 sockets
Jun 20 00:22:11 dist-test-slave-r81b krb5kdc[22370](info): commencing operation
krb5kdc: starting...
W20260620 00:22:12.576421 392 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:12 dist-test-slave-r81b krb5kdc[22370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914932, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:22:12.695081 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35265
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34435
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:12.776750 22383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:12.776993 22383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:12.777030 22383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:12.778190 22383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:12.778263 22383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:12.778287 22383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:12.778317 22383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:12.778358 22383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:12.779822 22383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34435
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35265
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:12.780098 22383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:12.780354 22383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:12.785199 22389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:12.785185 22391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:12.785319 22388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:12.785486 22383 server_base.cc:1034] running on GCE node
I20260620 00:22:12.785720 22383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:12.786043 22383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:12.787221 22383 hybrid_clock.cc:648] HybridClock initialized: now 1781914932787216 us; error 52 us; skew 500 ppm
Jun 20 00:22:12 dist-test-slave-r81b krb5kdc[22370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914932, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:12.790087 22383 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:22:12.790727 22383 webserver.cc:460] Webserver started at http://127.0.98.62:34683/ using document root <none> and password file <none>
I20260620 00:22:12.791038 22383 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:12.791091 22383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:12.791189 22383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:12.792378 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "3405a62129fd402a974c436d1df7ad5d"
format_stamp: "Formatted at 2026-06-20 00:22:12 on dist-test-slave-r81b"
server_key: "150e8ed1dd1fdd0a641ab528b09d6a03"
server_key_iv: "49c53088ad5577f9c4f47c1c10fa3469"
server_key_version: "encryptionkey@0"
I20260620 00:22:12.792707 22383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "3405a62129fd402a974c436d1df7ad5d"
format_stamp: "Formatted at 2026-06-20 00:22:12 on dist-test-slave-r81b"
server_key: "150e8ed1dd1fdd0a641ab528b09d6a03"
server_key_iv: "49c53088ad5577f9c4f47c1c10fa3469"
server_key_version: "encryptionkey@0"
I20260620 00:22:12.794168 22383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:22:12.795001 22398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:12.795209 22383 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:12.795331 22383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "3405a62129fd402a974c436d1df7ad5d"
format_stamp: "Formatted at 2026-06-20 00:22:12 on dist-test-slave-r81b"
server_key: "150e8ed1dd1fdd0a641ab528b09d6a03"
server_key_iv: "49c53088ad5577f9c4f47c1c10fa3469"
server_key_version: "encryptionkey@0"
I20260620 00:22:12.795420 22383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:12.838729 22383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:12.841128 22383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:12.841440 22383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:12.845340 22383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35265
I20260620 00:22:12.845372 22450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35265 every 8 connection(s)
I20260620 00:22:12.845719 22383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:12.846700 22451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:12.849020 22451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d: Bootstrap starting.
I20260620 00:22:12.849578 22451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:12.849817 22451 log.cc:824] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:12.850553 22451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d: No bootstrap required, opened a new log
I20260620 00:22:12.850920 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 22383
I20260620 00:22:12.851048 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:12.851257 392 external_mini_cluster.cc:1428] Setting key 3f24a4fbf735f7204e309f029ab74029
I20260620 00:22:12.853107 22451 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3405a62129fd402a974c436d1df7ad5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35265 } }
I20260620 00:22:12.853251 22451 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:12.853273 22451 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3405a62129fd402a974c436d1df7ad5d, State: Initialized, Role: FOLLOWER
I20260620 00:22:12.853402 22451 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3405a62129fd402a974c436d1df7ad5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35265 } }
I20260620 00:22:12.853475 22451 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:12.853519 22451 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:12.853569 22451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:12.854152 22451 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3405a62129fd402a974c436d1df7ad5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35265 } }
I20260620 00:22:12.854235 22451 leader_election.cc:302] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3405a62129fd402a974c436d1df7ad5d; no voters:
I20260620 00:22:12.854432 22451 leader_election.cc:288] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:12.854460 22456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:12.854684 22456 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [term 1 LEADER]: Becoming Leader. State: Replica: 3405a62129fd402a974c436d1df7ad5d, State: Running, Role: LEADER
I20260620 00:22:12.854707 22451 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:12.856175 22456 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3405a62129fd402a974c436d1df7ad5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35265 } }
I20260620 00:22:12.856778 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3405a62129fd402a974c436d1df7ad5d. Latest consensus state: current_term: 1 leader_uuid: "3405a62129fd402a974c436d1df7ad5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3405a62129fd402a974c436d1df7ad5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35265 } } }
I20260620 00:22:12.856845 22458 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3405a62129fd402a974c436d1df7ad5d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3405a62129fd402a974c436d1df7ad5d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35265 } } }
I20260620 00:22:12.857569 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:12.857585 22458 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:12.857894 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:12.851821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37287 (local address 127.0.98.62:35265)
0620 00:22:12.852050 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:12.852058 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:12.852083 (+ 25us) server_negotiation.cc:407] Connection header received
0620 00:22:12.852595 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:12.852602 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:12.852652 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:12.852772 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:12.853034 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:12.854463 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:12.855447 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:12.855941 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:12.856057 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:12.857093 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:12.857097 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:12.857097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:12.857325 (+ 228us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:12.857356 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:12.857368 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:12.857439 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:12.857609 (+ 170us) server_negotiation.cc:299] Negotiation successful
0620 00:22:12.857679 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":123,"threads_started":1}
W20260620 00:22:12.857213 22470 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:12.858002 22470 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:12.858071 22473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:12.858275 22473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:12.859061 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:12.860110 22473 catalog_manager.cc:1347] Generated new cluster ID: d3c926fc343f4c21b7472136b5fd3afd
I20260620 00:22:12.860191 22473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:12.868191 22455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:12.864559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54923 (local address 127.0.98.62:35265)
0620 00:22:12.864601 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:12.864603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:12.864728 (+ 125us) server_negotiation.cc:407] Connection header received
0620 00:22:12.864798 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:12.864799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:12.864809 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:12.864873 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:12.865050 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:12.865763 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:12.866836 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:12.867133 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:12.867195 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:12.867906 (+ 711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:12.867907 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:12.867907 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:12.867912 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:12.867936 (+ 24us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:12.867941 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:12.867993 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:12.868064 (+ 71us) server_negotiation.cc:299] Negotiation successful
0620 00:22:12.868085 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:22:12.883152 22473 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:12.883913 22473 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:12.890260 22473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3405a62129fd402a974c436d1df7ad5d: Generated new TSK 0
I20260620 00:22:12.890467 22473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:12.893937 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 22383
2026-06-20T00:22:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1395 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:12 dist-test-slave-r81b krb5kdc[22482](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:12 dist-test-slave-r81b krb5kdc[22482](info): set up 2 sockets
Jun 20 00:22:12 dist-test-slave-r81b krb5kdc[22482](info): commencing operation
krb5kdc: starting...
W20260620 00:22:13.952531 392 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:14 dist-test-slave-r81b krb5kdc[22482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914934, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:22:14.083763 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36619
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:40245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36619
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:14.169158 22495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:14.169451 22495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:14.169498 22495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:14.170795 22495 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:14.170877 22495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:14.170903 22495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:14.170933 22495 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:14.170964 22495 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:14.172626 22495 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:40245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36619
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36619
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:14.172933 22495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:14.173177 22495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:14.178203 22501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:14.178180 22503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:14.178180 22500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:14.178483 22495 server_base.cc:1034] running on GCE node
I20260620 00:22:14.178737 22495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:14.179037 22495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:14.180214 22495 hybrid_clock.cc:648] HybridClock initialized: now 1781914934180195 us; error 32 us; skew 500 ppm
Jun 20 00:22:14 dist-test-slave-r81b krb5kdc[22482](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914934, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:14.182705 22495 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:22:14.183192 22495 webserver.cc:460] Webserver started at http://127.0.98.62:34065/ using document root <none> and password file <none>
I20260620 00:22:14.183441 22495 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:14.183511 22495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:14.183661 22495 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:14.184805 22495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "d8a84c4f287e49db8d091b6efd17c5b3"
format_stamp: "Formatted at 2026-06-20 00:22:14 on dist-test-slave-r81b"
server_key: "f9796338c05afb6501216e88aadd6174"
server_key_iv: "1cafb1aa7eabc6b4201f17b94ed5832b"
server_key_version: "encryptionkey@0"
I20260620 00:22:14.185177 22495 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "d8a84c4f287e49db8d091b6efd17c5b3"
format_stamp: "Formatted at 2026-06-20 00:22:14 on dist-test-slave-r81b"
server_key: "f9796338c05afb6501216e88aadd6174"
server_key_iv: "1cafb1aa7eabc6b4201f17b94ed5832b"
server_key_version: "encryptionkey@0"
I20260620 00:22:14.186532 22495 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:22:14.187351 22510 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:14.187542 22495 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:14.187680 22495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "d8a84c4f287e49db8d091b6efd17c5b3"
format_stamp: "Formatted at 2026-06-20 00:22:14 on dist-test-slave-r81b"
server_key: "f9796338c05afb6501216e88aadd6174"
server_key_iv: "1cafb1aa7eabc6b4201f17b94ed5832b"
server_key_version: "encryptionkey@0"
I20260620 00:22:14.187763 22495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:14.191187 22495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:14.193378 22495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:14.193537 22495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:14.197962 22495 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36619
I20260620 00:22:14.198011 22562 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36619 every 8 connection(s)
I20260620 00:22:14.198380 22495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:14.198858 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 22495
I20260620 00:22:14.199005 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:14.199268 392 external_mini_cluster.cc:1428] Setting key d3534912ea70d14f2b0b44a280f74b5e
I20260620 00:22:14.199713 22563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:14.201822 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:14.199835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42453 (local address 127.0.98.62:36619)
0620 00:22:14.200035 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:14.200043 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:14.200074 (+ 31us) server_negotiation.cc:407] Connection header received
0620 00:22:14.200561 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:14.200569 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:14.200594 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:14.200711 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:14.201356 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:14.201359 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:14.201359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:14.201363 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:14.201404 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:14.201415 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:14.201470 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:14.201568 (+ 98us) server_negotiation.cc:299] Negotiation successful
0620 00:22:14.201626 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":96,"threads_started":1}
I20260620 00:22:14.203059 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:14.203501 22563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3: Bootstrap starting.
I20260620 00:22:14.204401 22563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:14.204823 22563 log.cc:824] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:14.205801 22563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3: No bootstrap required, opened a new log
I20260620 00:22:14.207147 22566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:14.205444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45849 (local address 127.0.98.62:36619)
0620 00:22:14.205501 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:14.205503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:14.205914 (+ 411us) server_negotiation.cc:407] Connection header received
0620 00:22:14.206056 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:14.206057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:14.206063 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:14.206095 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:14.206877 (+ 782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:14.206878 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:14.206879 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0620 00:22:14.206884 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:14.206901 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0620 00:22:14.206906 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:14.206962 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:14.207045 (+ 83us) server_negotiation.cc:299] Negotiation successful
0620 00:22:14.207061 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260620 00:22:14.208293 22563 raft_consensus.cc:348] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a84c4f287e49db8d091b6efd17c5b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36619 } }
I20260620 00:22:14.208493 22563 raft_consensus.cc:374] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260620 00:22:14.208544 22563 raft_consensus.cc:729] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8a84c4f287e49db8d091b6efd17c5b3, State: Initialized, Role: FOLLOWER
I20260620 00:22:14.208671 22563 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a84c4f287e49db8d091b6efd17c5b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36619 } }
I20260620 00:22:14.208776 22563 raft_consensus.cc:388] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260620 00:22:14.208819 22563 raft_consensus.cc:482] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260620 00:22:14.208869 22563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:14.209439 22563 raft_consensus.cc:504] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a84c4f287e49db8d091b6efd17c5b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36619 } }
I20260620 00:22:14.209565 22563 leader_election.cc:302] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8a84c4f287e49db8d091b6efd17c5b3; no voters:
I20260620 00:22:14.209770 22563 leader_election.cc:288] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260620 00:22:14.209884 22573 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:14.210026 22563 sys_catalog.cc:564] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:14.210153 22573 raft_consensus.cc:686] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [term 1 LEADER]: Becoming Leader. State: Replica: d8a84c4f287e49db8d091b6efd17c5b3, State: Running, Role: LEADER
I20260620 00:22:14.210273 22573 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a84c4f287e49db8d091b6efd17c5b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36619 } }
I20260620 00:22:14.210690 22574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8a84c4f287e49db8d091b6efd17c5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a84c4f287e49db8d091b6efd17c5b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36619 } } }
I20260620 00:22:14.210772 22574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:14.210937 22574 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8a84c4f287e49db8d091b6efd17c5b3. Latest consensus state: current_term: 1 leader_uuid: "d8a84c4f287e49db8d091b6efd17c5b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8a84c4f287e49db8d091b6efd17c5b3" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36619 } } }
I20260620 00:22:14.210995 22574 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:14.211025 22582 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:14.211588 22582 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:14.213189 22582 catalog_manager.cc:1347] Generated new cluster ID: 7f177f3f68ab4fc19a7f56c21edf4bf8
I20260620 00:22:14.213240 22582 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:14.218677 22582 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:14.218879 22582 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:14.226389 22582 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d8a84c4f287e49db8d091b6efd17c5b3: Generated new TSK 0
I20260620 00:22:14.226624 22582 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:14.235082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 22495
2026-06-20T00:22:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1339 ms)
[----------] 18 tests from AuthTokenIssuingTest (24545 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:14 dist-test-slave-r81b krb5kdc[22594](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:14 dist-test-slave-r81b krb5kdc[22594](info): set up 2 sockets
Jun 20 00:22:14 dist-test-slave-r81b krb5kdc[22594](info): commencing operation
krb5kdc: starting...
W20260620 00:22:15.328516 392 mini_kdc.cc:121] Time spent starting KDC: real 1.073s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:22:15.452064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46673
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46673,127.0.98.61:46591,127.0.98.60:40501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:15.536068 22607 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:22:15.536353 22607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:15.536372 22607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:15.536393 22607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:15.537629 22607 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:15.537674 22607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:15.537689 22607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:15.537701 22607 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:15.537714 22607 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:15.539291 22607 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46673,127.0.98.61:46591,127.0.98.60:40501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22607
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:15.539579 22607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:15.539822 22607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:15.544979 22613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:15.545109 22607 server_base.cc:1034] running on GCE node
W20260620 00:22:15.545027 22615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:15.544986 22612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:15.545473 22607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:15.545758 22607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:15.546927 22607 hybrid_clock.cc:648] HybridClock initialized: now 1781914935546909 us; error 49 us; skew 500 ppm
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:15.554589 22607 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:22:15.555116 22607 webserver.cc:460] Webserver started at http://127.0.98.62:36345/ using document root <none> and password file <none>
I20260620 00:22:15.555358 22607 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:15.555408 22607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:15.555545 22607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:15.556740 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "544cb78f49a847738272a54619aff670"
format_stamp: "Formatted at 2026-06-20 00:22:15 on dist-test-slave-r81b"
server_key: "0090703b09407d1b8a4c4dd14bd752f8"
server_key_iv: "e10563af89c703f7bdce88ec52e8195f"
server_key_version: "encryptionkey@0"
I20260620 00:22:15.557142 22607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "544cb78f49a847738272a54619aff670"
format_stamp: "Formatted at 2026-06-20 00:22:15 on dist-test-slave-r81b"
server_key: "0090703b09407d1b8a4c4dd14bd752f8"
server_key_iv: "e10563af89c703f7bdce88ec52e8195f"
server_key_version: "encryptionkey@0"
I20260620 00:22:15.558620 22607 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:22:15.559432 22622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:15.559641 22607 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:15.559727 22607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "544cb78f49a847738272a54619aff670"
format_stamp: "Formatted at 2026-06-20 00:22:15 on dist-test-slave-r81b"
server_key: "0090703b09407d1b8a4c4dd14bd752f8"
server_key_iv: "e10563af89c703f7bdce88ec52e8195f"
server_key_version: "encryptionkey@0"
I20260620 00:22:15.559784 22607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:15.571266 22607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:15.573628 22607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:15.573745 22607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:15.577759 22607 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46673
I20260620 00:22:15.577853 22674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46673 every 8 connection(s)
I20260620 00:22:15.578186 22607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:15.579128 22675 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:15.580832 22675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:22:15.587462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 22607
I20260620 00:22:15.587577 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:15.587771 392 external_mini_cluster.cc:1428] Setting key 2aba5a11236a5731a06667fb61fd78d2
I20260620 00:22:15.591646 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.581639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35090 (local address 127.0.98.62:46673)
0620 00:22:15.581826 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.581829 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.581865 (+ 36us) server_negotiation.cc:407] Connection header received
0620 00:22:15.582360 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.582362 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.582378 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.582463 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.583056 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.584202 (+ 1146us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.585114 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.585466 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.585553 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.587844 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.587851 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.587853 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.587860 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.589905 (+ 2045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.590402 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.590405 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.590406 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.590489 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.590753 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.590754 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.590754 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.590975 (+ 221us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.591040 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.591252 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.591380 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":148,"thread_start_us":95,"threads_started":1}
I20260620 00:22:15.591658 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.581553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46673 (local address 127.0.0.1:35090)
0620 00:22:15.581713 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.581764 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:15.582229 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:15.582505 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:15.582508 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:15.582765 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:15.582936 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.582940 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.584304 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.584305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.585029 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.585031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.585579 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.585580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.585640 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.586171 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:15.586180 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:15.587744 (+ 1564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:15.590002 (+ 2258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.590009 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.590011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.590287 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.590567 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.590568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.590569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.590685 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.591046 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:15.591052 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:15.591129 (+ 77us) client_negotiation.cc:769] Sending connection context
0620 00:22:15.591237 (+ 108us) client_negotiation.cc:240] Negotiation successful
0620 00:22:15.591465 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":65,"thread_start_us":67,"threads_started":1}
I20260620 00:22:15.592859 22675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:22:15.593276 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.593128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46591 (local address 127.0.0.1:38000)
0620 00:22:15.593184 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.593238 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:22:15.593452 22624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
I20260620 00:22:15.593945 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.593830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46591 (local address 127.0.0.1:38004)
0620 00:22:15.593880 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.593913 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:22:15.594076 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46591: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 00:22:15.616690 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46591
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46673,127.0.98.61:46591,127.0.98.60:40501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:22:15.643349 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } attempt: 1
I20260620 00:22:15.643937 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.643788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46591 (local address 127.0.0.1:38010)
0620 00:22:15.643856 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.643904 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260620 00:22:15.644351 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.644248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46591 (local address 127.0.0.1:38026)
0620 00:22:15.644294 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.644332 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 00:22:15.644443 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46591: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
W20260620 00:22:15.703517 22682 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:22:15.703742 22682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:15.703759 22682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:15.703780 22682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:15.705045 22682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:15.705092 22682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:15.705107 22682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:15.705120 22682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:15.705133 22682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:15.706717 22682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46673,127.0.98.61:46591,127.0.98.60:40501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:46591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:15.706997 22682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:15.707242 22682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:15.712365 22690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:15.712390 22687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:15.712563 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } attempt: 2
I20260620 00:22:15.712764 22682 server_base.cc:1034] running on GCE node
W20260620 00:22:15.712374 22688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:15.712951 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.712813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46591 (local address 127.0.0.1:38028)
0620 00:22:15.712864 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.712897 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260620 00:22:15.713119 22682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:15.713313 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.713222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46591 (local address 127.0.0.1:38044)
0620 00:22:15.713269 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.713294 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 00:22:15.713405 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:46591: Network error: Client connection negotiation failed: client connection to 127.0.98.61:46591: connect: Connection refused (error 111)
I20260620 00:22:15.713575 22682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:15.714756 22682 hybrid_clock.cc:648] HybridClock initialized: now 1781914935714747 us; error 35 us; skew 500 ppm
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:15.717398 22682 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 00:22:15.717880 22682 webserver.cc:460] Webserver started at http://127.0.98.61:35003/ using document root <none> and password file <none>
I20260620 00:22:15.718115 22682 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:15.718185 22682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:15.718338 22682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:15.719412 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/data/instance:
uuid: "fcd1c0f3673744999377efd91936d7e7"
format_stamp: "Formatted at 2026-06-20 00:22:15 on dist-test-slave-r81b"
server_key: "f0b98606e9c26b9d0a11326d84dffbf7"
server_key_iv: "3d5b65b1c3356ef5382be8900cac7c86"
server_key_version: "encryptionkey@0"
I20260620 00:22:15.719800 22682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/wal/instance:
uuid: "fcd1c0f3673744999377efd91936d7e7"
format_stamp: "Formatted at 2026-06-20 00:22:15 on dist-test-slave-r81b"
server_key: "f0b98606e9c26b9d0a11326d84dffbf7"
server_key_iv: "3d5b65b1c3356ef5382be8900cac7c86"
server_key_version: "encryptionkey@0"
I20260620 00:22:15.721282 22682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260620 00:22:15.722149 22697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:15.722324 22682 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260620 00:22:15.722476 22682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/wal
uuid: "fcd1c0f3673744999377efd91936d7e7"
format_stamp: "Formatted at 2026-06-20 00:22:15 on dist-test-slave-r81b"
server_key: "f0b98606e9c26b9d0a11326d84dffbf7"
server_key_iv: "3d5b65b1c3356ef5382be8900cac7c86"
server_key_version: "encryptionkey@0"
I20260620 00:22:15.722551 22682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:15.733387 22682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:15.735620 22682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:15.735795 22682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:15.739892 22682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:46591
I20260620 00:22:15.739924 22749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:46591 every 8 connection(s)
I20260620 00:22:15.740306 22682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:22:15.741323 22750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:15.742221 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 22682
I20260620 00:22:15.742336 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-1/wal/instance
I20260620 00:22:15.742501 392 external_mini_cluster.cc:1428] Setting key da93ac2cc3e841b7203b1847aef5d1dd
I20260620 00:22:15.743062 22750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260620 00:22:15.752318 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.743774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35092 (local address 127.0.98.62:46673)
0620 00:22:15.743823 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.743825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.744085 (+ 260us) server_negotiation.cc:407] Connection header received
0620 00:22:15.744638 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.744640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.744656 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.744710 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.745342 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.745946 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.746945 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.747170 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.747258 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.749442 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.749444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.749444 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.749452 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.750877 (+ 1425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.751291 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.751293 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.751293 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.751357 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.751620 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.751621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.751621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.751741 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.751814 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.752116 (+ 302us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.752226 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:22:15.752516 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.743773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46673 (local address 127.0.0.1:35092)
0620 00:22:15.743925 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.743975 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:15.744528 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:15.744789 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:15.744793 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:15.745048 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:15.745243 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.745247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.746046 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.746047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.746841 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.746843 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.747279 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.747280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.747414 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.747951 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:15.747959 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:15.749343 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:15.750987 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.750994 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.750996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.751213 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.751425 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.751425 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.751426 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.751549 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.751818 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:15.751824 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:15.751980 (+ 156us) client_negotiation.cc:769] Sending connection context
0620 00:22:15.752083 (+ 103us) client_negotiation.cc:240] Negotiation successful
0620 00:22:15.752267 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":80,"threads_started":1}
I20260620 00:22:15.753113 22750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:22:15.760882 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.753392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46591 (local address 127.0.0.1:38046)
0620 00:22:15.753434 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.753443 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:15.753476 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:15.753768 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:15.753769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:15.753905 (+ 136us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:15.753956 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.753957 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.754793 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.754794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.755460 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.755462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.755948 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.755949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.756014 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.756445 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:15.756450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:15.757704 (+ 1254us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:15.759615 (+ 1911us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.759616 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.759617 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.759864 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.760072 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.760073 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.760073 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.760181 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.760600 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:15.760601 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:15.760657 (+ 56us) client_negotiation.cc:769] Sending connection context
0620 00:22:15.760729 (+ 72us) client_negotiation.cc:240] Negotiation successful
0620 00:22:15.760813 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260620 00:22:15.761070 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.753475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.0.98.61:46591)
0620 00:22:15.753639 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.753642 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.753658 (+ 16us) server_negotiation.cc:407] Connection header received
0620 00:22:15.753674 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.753675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.753685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.753768 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.754030 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.754705 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.755544 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.755854 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.755944 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.757814 (+ 1870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.757822 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.757823 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.757829 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.759511 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.759952 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.759955 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.759956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.760009 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.760233 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.760233 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.760234 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.760374 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.760487 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.760770 (+ 283us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.760876 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260620 00:22:15.761672 22750 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:22:15.762095 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.761956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45238)
0620 00:22:15.762008 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.762059 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:22:15.762239 22699 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
I20260620 00:22:15.762671 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.762576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45244)
0620 00:22:15.762621 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.762647 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:22:15.762782 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 00:22:15.767786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40501
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36469
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46673,127.0.98.61:46591,127.0.98.60:40501
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:22:15.804571 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } attempt: 3
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:22:15.811995 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } attempt: 1
I20260620 00:22:15.812270 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.804991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46591 (local address 127.0.0.1:38052)
0620 00:22:15.805088 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.805103 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:15.805159 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:15.805316 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:15.805317 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:15.805487 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:15.805567 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.805569 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.806278 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.806280 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.806825 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.806827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.807215 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.807216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.807298 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.807783 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:15.807788 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:15.809073 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:15.810957 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.810958 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.810958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.811145 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.811405 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.811413 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.811413 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.811514 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.811864 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:15.811866 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:15.811924 (+ 58us) client_negotiation.cc:769] Sending connection context
0620 00:22:15.812013 (+ 89us) client_negotiation.cc:240] Negotiation successful
0620 00:22:15.812134 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260620 00:22:15.812269 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.805081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.0.98.61:46591)
0620 00:22:15.805155 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.805157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.805169 (+ 12us) server_negotiation.cc:407] Connection header received
0620 00:22:15.805223 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.805224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.805236 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.805308 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.805678 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.806189 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.806914 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.807137 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.807225 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.809170 (+ 1945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.809172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.809172 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.809182 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.810848 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.811226 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.811227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.811228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.811300 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.811627 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.811628 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.811628 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.811775 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.811863 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.812041 (+ 178us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.812174 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:22:15.812809 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.812311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45252)
0620 00:22:15.812739 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.812780 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407}
I20260620 00:22:15.813058 22675 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:22:15.813243 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.813125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45254)
0620 00:22:15.813178 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.813209 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260620 00:22:15.813351 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
I20260620 00:22:15.813446 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.813290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45264)
0620 00:22:15.813373 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.813414 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260620 00:22:15.813781 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.813721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45268)
0620 00:22:15.813743 (+ 22us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.813765 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 00:22:15.813886 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
I20260620 00:22:15.845052 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } attempt: 1
I20260620 00:22:15.845638 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.845473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45274)
0620 00:22:15.845560 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.845605 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260620 00:22:15.846047 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.845960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45284)
0620 00:22:15.846002 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.846029 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:22:15.846180 22675 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
W20260620 00:22:15.852596 22757 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:22:15.852807 22757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:15.852828 22757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:15.852849 22757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:15.853978 22757 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:15.854014 22757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:15.854028 22757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:15.854041 22757 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:15.854053 22757 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:15.855614 22757 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36469
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46673,127.0.98.61:46591,127.0.98.60:40501
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:40501
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22757
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:15.855818 22757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:15.856002 22757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:15.861088 22763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:15.861100 22762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:15.861112 22765 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:15.861395 22757 server_base.cc:1034] running on GCE node
I20260620 00:22:15.861680 22757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:15.862001 22757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:15.863196 22757 hybrid_clock.cc:648] HybridClock initialized: now 1781914935863189 us; error 47 us; skew 500 ppm
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:15.865980 22757 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 00:22:15.866483 22757 webserver.cc:460] Webserver started at http://127.0.98.60:38937/ using document root <none> and password file <none>
I20260620 00:22:15.866752 22757 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:15.866823 22757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:15.866947 22757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:15.868014 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/data/instance:
uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809"
format_stamp: "Formatted at 2026-06-20 00:22:15 on dist-test-slave-r81b"
server_key: "5e74d0a5dc4e2732ec4d27633e9e7663"
server_key_iv: "b7ef2a3f0a671f1da7716f37d9e52f9e"
server_key_version: "encryptionkey@0"
I20260620 00:22:15.868402 22757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/wal/instance:
uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809"
format_stamp: "Formatted at 2026-06-20 00:22:15 on dist-test-slave-r81b"
server_key: "5e74d0a5dc4e2732ec4d27633e9e7663"
server_key_iv: "b7ef2a3f0a671f1da7716f37d9e52f9e"
server_key_version: "encryptionkey@0"
I20260620 00:22:15.869709 22757 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:15.870393 22772 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:15.870622 22757 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:15.870697 22757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/wal
uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809"
format_stamp: "Formatted at 2026-06-20 00:22:15 on dist-test-slave-r81b"
server_key: "5e74d0a5dc4e2732ec4d27633e9e7663"
server_key_iv: "b7ef2a3f0a671f1da7716f37d9e52f9e"
server_key_version: "encryptionkey@0"
I20260620 00:22:15.870795 22757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:15.881496 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } attempt: 2
I20260620 00:22:15.881999 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.881863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45290)
0620 00:22:15.881921 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.881965 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 00:22:15.882402 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.882325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45296)
0620 00:22:15.882363 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.882386 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260620 00:22:15.882541 22750 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:40501: Network error: Client connection negotiation failed: client connection to 127.0.98.60:40501: connect: Connection refused (error 111)
I20260620 00:22:15.895629 22757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:15.898036 22757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:15.898197 22757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:15.902534 22757 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:40501
I20260620 00:22:15.902580 22824 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:40501 every 8 connection(s)
I20260620 00:22:15.903016 22757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:22:15.904071 22825 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:15.905741 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:22:15.913173 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 22757
I20260620 00:22:15.913291 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1781914666918284-392-0/minicluster-data/master-2/wal/instance
I20260620 00:22:15.913524 392 external_mini_cluster.cc:1428] Setting key 745efa8ff6640d18c6670d4914b45c49
I20260620 00:22:15.915261 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.906514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35096 (local address 127.0.98.62:46673)
0620 00:22:15.906630 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.906633 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.906809 (+ 176us) server_negotiation.cc:407] Connection header received
0620 00:22:15.907332 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.907334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.907347 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.907405 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.908048 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.908589 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.909690 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.909901 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.909994 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.912278 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.912279 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.912280 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.912289 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.913801 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.914300 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.914301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.914302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.914367 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.914599 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.914600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.914600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.914750 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.914828 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.915082 (+ 254us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.915172 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260620 00:22:15.915441 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.906428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46673 (local address 127.0.0.1:35096)
0620 00:22:15.906669 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.906716 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:15.907219 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:15.907496 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:15.907499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:15.907748 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:15.907950 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.907954 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.908702 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.908703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.909581 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.909584 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.910018 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.910019 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.910150 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.910627 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:15.910642 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:15.912107 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:15.913895 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.913903 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.913905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.914241 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.914430 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.914430 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.914431 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.914532 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.914821 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:15.914825 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:15.914948 (+ 123us) client_negotiation.cc:769] Sending connection context
0620 00:22:15.915047 (+ 99us) client_negotiation.cc:240] Negotiation successful
0620 00:22:15.915214 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":116,"threads_started":1}
I20260620 00:22:15.916157 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:22:15.921398 22675 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } attempt: 2
I20260620 00:22:15.923714 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.913926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45304 (local address 127.0.98.60:40501)
0620 00:22:15.914165 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.914170 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.914193 (+ 23us) server_negotiation.cc:407] Connection header received
0620 00:22:15.914260 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.914263 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.914278 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.914353 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.915181 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.916090 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.917133 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.917433 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.917488 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.920103 (+ 2615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.920111 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.920111 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.920118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.922101 (+ 1983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.922463 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.922466 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.922467 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.922533 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.922736 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.922737 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.922737 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.922956 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.923109 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.923523 (+ 414us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.923600 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":194,"thread_start_us":93,"threads_started":1}
I20260620 00:22:15.923940 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.916498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38062 (local address 127.0.98.61:46591)
0620 00:22:15.916551 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.916553 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.916562 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:22:15.916589 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.916590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.916601 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.916658 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.916901 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.917458 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.918402 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.918752 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.918831 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.920803 (+ 1972us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.920805 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.920805 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.920813 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.922548 (+ 1735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.923020 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.923022 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.923022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.923104 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.923323 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.923324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.923324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.923479 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.923557 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.923756 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.923845 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:22:15.923940 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.916418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:46591 (local address 127.0.0.1:38062)
0620 00:22:15.916469 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.916482 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:15.916523 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:15.916669 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:15.916669 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:15.916785 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:15.916830 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.916831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.917561 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.917562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.918313 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.918315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.918835 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.918835 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.918884 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.919294 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:15.919299 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:15.920710 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:15.922660 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.922661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.922662 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.922915 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.923178 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.923179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.923180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.923251 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.923589 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:15.923590 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:15.923636 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 00:22:15.923729 (+ 93us) client_negotiation.cc:240] Negotiation successful
0620 00:22:15.923834 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260620 00:22:15.924772 22825 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:22:15.931491 22677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.921733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45306)
0620 00:22:15.921781 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.921814 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:15.921876 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:15.922874 (+ 998us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:15.922876 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:15.923051 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:15.923100 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.923103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.924020 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.924022 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.924798 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.924801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.925372 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.925373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.925443 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.925925 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:15.925930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:15.927567 (+ 1637us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:15.930366 (+ 2799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.930367 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.930367 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.930597 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.930806 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.930807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.930807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.930875 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.931143 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:15.931144 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:15.931188 (+ 44us) client_negotiation.cc:769] Sending connection context
0620 00:22:15.931270 (+ 82us) client_negotiation.cc:240] Negotiation successful
0620 00:22:15.931378 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260620 00:22:15.931522 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.922027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45306 (local address 127.0.98.60:40501)
0620 00:22:15.922728 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.922730 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.922738 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:22:15.922758 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.922759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.922772 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.922854 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.923182 (+ 328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.923924 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.924892 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.925237 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.925612 (+ 375us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.927677 (+ 2065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.927678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.927679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.927686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.930275 (+ 2589us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.930690 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.930692 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.930692 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.930753 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.930947 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.930947 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.930948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.931064 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.931137 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.931295 (+ 158us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.931400 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":124,"threads_started":1}
I20260620 00:22:15.933148 22827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.925081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45322)
0620 00:22:15.925173 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.925182 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:15.925213 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:15.925411 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:15.925412 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:15.925578 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:15.925621 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.925623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.926352 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.926352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.927192 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.927194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.927938 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.927939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.927985 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.928417 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:15.928422 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:15.929617 (+ 1195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:15.931800 (+ 2183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.931800 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.931801 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.932252 (+ 451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.932460 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.932460 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.932461 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.932516 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.932792 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:15.932793 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:15.932825 (+ 32us) client_negotiation.cc:769] Sending connection context
0620 00:22:15.932893 (+ 68us) client_negotiation.cc:240] Negotiation successful
0620 00:22:15.933010 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260620 00:22:15.933244 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.925149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45322 (local address 127.0.98.60:40501)
0620 00:22:15.925289 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.925290 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.925296 (+ 6us) server_negotiation.cc:407] Connection header received
0620 00:22:15.925317 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.925317 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.925327 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.925406 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.925709 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.926260 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.927281 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.927578 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.927666 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.929716 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.929717 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.929717 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.929724 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.931713 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.932330 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.932331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.932332 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.932384 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.932576 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.932577 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.932577 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.932717 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.932772 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.932937 (+ 165us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.933071 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":47,"server-negotiator.queue_time_us":111}
I20260620 00:22:15.933804 22675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670: Bootstrap starting.
I20260620 00:22:15.934062 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.924914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35112 (local address 127.0.98.62:46673)
0620 00:22:15.924961 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.924963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.925015 (+ 52us) server_negotiation.cc:407] Connection header received
0620 00:22:15.925092 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.925093 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.925105 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.925168 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.925850 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.926516 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.928079 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.928391 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.928458 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.931166 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.931167 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.931168 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.931175 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.932689 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.933109 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.933111 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.933111 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.933168 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.933375 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.933376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.933377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.933523 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.933686 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.933885 (+ 199us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.933962 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260620 00:22:15.934643 22675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:15.934952 22675 log.cc:824] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:15.935807 22675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670: No bootstrap required, opened a new log
I20260620 00:22:15.936442 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809: Bootstrap starting.
I20260620 00:22:15.937089 22825 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:15.937414 22825 log.cc:824] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:15.938020 22675 raft_consensus.cc:348] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } }
I20260620 00:22:15.938180 22825 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809: No bootstrap required, opened a new log
I20260620 00:22:15.938189 22675 raft_consensus.cc:729] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 544cb78f49a847738272a54619aff670, State: Initialized, Role: FOLLOWER
I20260620 00:22:15.938280 22675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } }
I20260620 00:22:15.938573 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:15.938639 22675 sys_catalog.cc:564] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:15.938658 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:22:15.940119 22825 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } }
I20260620 00:22:15.940529 22825 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0b7a4d95f4c41bcb0effd3f47e9c809, State: Initialized, Role: FOLLOWER
I20260620 00:22:15.940667 22825 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } }
W20260620 00:22:15.940770 22844 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:15.940846 22844 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:15.940979 22825 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:15.942494 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:15.943449 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [sys.catalog]: This master's current role is: FOLLOWER
W20260620 00:22:15.943179 22857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:15.944211 22857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:15.944530 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.934707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38078 (local address 127.0.98.61:46591)
0620 00:22:15.934749 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.934751 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.934790 (+ 39us) server_negotiation.cc:407] Connection header received
0620 00:22:15.934866 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.934867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.934877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.934931 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.935527 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.936255 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.937238 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.937549 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.937596 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.940872 (+ 3276us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.940873 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.940873 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.940883 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.942814 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.943305 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.943307 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.943307 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.943368 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.943649 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.943649 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.943650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.943787 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.943924 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.944350 (+ 426us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.944429 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:22:15.941989 22607 master_runner.cc:186] Error getting master registration for 127.0.98.61:46591: OK, instance_id { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" instance_seqno: 1781914935738788 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:22:15.944902 22607 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:22:15.945672 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:15.946697 22649 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "544cb78f49a847738272a54619aff670"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35112
I20260620 00:22:15.946820 22649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 00:22:15.946866 22649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:15.947786 22649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } }
I20260620 00:22:15.947988 22649 leader_election.cc:288] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [CANDIDATE]: Term 1 election: Requested vote from peers fcd1c0f3673744999377efd91936d7e7 (127.0.98.61:46591), d0b7a4d95f4c41bcb0effd3f47e9c809 (127.0.98.60:40501)
I20260620 00:22:15.948477 22799 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "544cb78f49a847738272a54619aff670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809"
I20260620 00:22:15.948602 22799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:15.946655 22757 master_runner.cc:186] Error getting master registration for 127.0.98.61:46591: OK, instance_id { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" instance_seqno: 1781914935738788 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:22:15.949095 22757 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:22:15.949543 22799 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544cb78f49a847738272a54619aff670 in term 1.
I20260620 00:22:15.949744 22624 leader_election.cc:302] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 544cb78f49a847738272a54619aff670, d0b7a4d95f4c41bcb0effd3f47e9c809; no voters:
I20260620 00:22:15.949898 22833 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:15.950100 22833 raft_consensus.cc:686] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [term 1 LEADER]: Becoming Leader. State: Replica: 544cb78f49a847738272a54619aff670, State: Running, Role: LEADER
I20260620 00:22:15.950201 22833 consensus_queue.cc:237] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } }
I20260620 00:22:15.950700 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 544cb78f49a847738272a54619aff670. Latest consensus state: current_term: 1 leader_uuid: "544cb78f49a847738272a54619aff670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:15.950915 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:15.951293 22865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:15.951503 22865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:15.953591 22799 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [term 1 FOLLOWER]: Refusing update from remote peer 544cb78f49a847738272a54619aff670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:22:15.954052 22863 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:22:15.955542 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 544cb78f49a847738272a54619aff670. Latest consensus state: current_term: 1 leader_uuid: "544cb78f49a847738272a54619aff670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:15.955765 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:15.964741 22863 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "544cb78f49a847738272a54619aff670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:15.964872 22863 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:15.965260 22846 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "544cb78f49a847738272a54619aff670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:15.965358 22846 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:15.967013 22865 catalog_manager.cc:1347] Generated new cluster ID: 3458ca1819b447f1b547c374d3e0f69a
I20260620 00:22:15.967073 22865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:15.973659 22750 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } attempt: 3
Jun 20 00:22:15 dist-test-slave-r81b krb5kdc[22594](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914935, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:22:15.980396 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.956379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35114 (local address 127.0.98.62:46673)
0620 00:22:15.956451 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.956453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.957027 (+ 574us) server_negotiation.cc:407] Connection header received
0620 00:22:15.957205 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.957207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.957218 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.957250 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.958794 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.959557 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.961681 (+ 2124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.962015 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.962058 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.970225 (+ 8167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.970227 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.970227 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.970235 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.972520 (+ 2285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.977276 (+ 4756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.977277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.977278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.977344 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.978974 (+ 1630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.978976 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.978976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.979123 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.979211 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.980200 (+ 989us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.980291 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260620 00:22:15.981493 22753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.973926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:40501 (local address 127.0.0.1:45344)
0620 00:22:15.974006 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:15.974016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:15.974049 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:15.974299 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:15.974300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:15.974461 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:15.974503 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.974505 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.975388 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.975390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.976172 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:15.976173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.976691 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.976692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:15.976743 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.977119 (+ 376us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:15.977124 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:15.978435 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:15.980564 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.980565 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.980566 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.980752 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.980917 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:15.980918 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:15.980918 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:15.980972 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:15.981209 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:15.981210 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:15.981249 (+ 39us) client_negotiation.cc:769] Sending connection context
0620 00:22:15.981301 (+ 52us) client_negotiation.cc:240] Negotiation successful
0620 00:22:15.981398 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260620 00:22:15.981802 22830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.974132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45344 (local address 127.0.98.60:40501)
0620 00:22:15.974168 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.974170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.974177 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:22:15.974209 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.974209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.974220 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.974260 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.974571 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.975301 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.976261 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.976583 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.976640 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.978518 (+ 1878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.978520 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.978520 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.978527 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.980480 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.980814 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.980815 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.980815 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.980863 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.981030 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.981030 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.981031 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.981142 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.981190 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.981650 (+ 460us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.981722 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260620 00:22:15.983016 22755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.956592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38082 (local address 127.0.98.61:46591)
0620 00:22:15.956633 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.956635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.960358 (+ 3723us) server_negotiation.cc:407] Connection header received
0620 00:22:15.960461 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.960463 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.960475 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.960505 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.961415 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.962178 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.963857 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.964205 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.964251 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.974174 (+ 9923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.974176 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.974176 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.974184 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.976528 (+ 2344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.979639 (+ 3111us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.979640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.979641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.979714 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.981377 (+ 1663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.981391 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.981392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.981530 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.981617 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.982843 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.982926 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:22:15.983251 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7: Bootstrap starting.
I20260620 00:22:15.983976 22750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:15.984298 22750 log.cc:824] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:15.997144 22750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7: No bootstrap required, opened a new log
I20260620 00:22:15.997659 22829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:15.956746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45334 (local address 127.0.98.60:40501)
0620 00:22:15.956796 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:15.956798 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:15.961352 (+ 4554us) server_negotiation.cc:407] Connection header received
0620 00:22:15.961400 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:15.961401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:15.961413 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:15.961469 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:15.966149 (+ 4680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.966887 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.977646 (+ 10759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:15.977957 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:15.977996 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:15.985241 (+ 7245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:15.985243 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:15.985243 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:15.985250 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:15.986835 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.996687 (+ 9852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.996689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.996690 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.996779 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:15.997005 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:15.997006 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:15.997007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:15.997142 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:15.997231 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:15.997473 (+ 242us) server_negotiation.cc:299] Negotiation successful
0620 00:22:15.997573 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260620 00:22:15.998102 22865 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:15.999526 22865 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:15.999471 22750 raft_consensus.cc:348] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } }
I20260620 00:22:15.999617 22750 raft_consensus.cc:729] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcd1c0f3673744999377efd91936d7e7, State: Initialized, Role: FOLLOWER
I20260620 00:22:15.999708 22750 consensus_queue.cc:260] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } }
I20260620 00:22:16.000094 22750 sys_catalog.cc:564] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:16.000159 22724 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "544cb78f49a847738272a54619aff670" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "fcd1c0f3673744999377efd91936d7e7"
I20260620 00:22:16.000288 22724 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:16.001204 22724 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 544cb78f49a847738272a54619aff670 in term 1.
I20260620 00:22:16.001602 22723 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [term 1 FOLLOWER]: Refusing update from remote peer 544cb78f49a847738272a54619aff670: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:22:16.002873 22833 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260620 00:22:16.005528 22886 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:16.005595 22886 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:16.011729 22887 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 544cb78f49a847738272a54619aff670. Latest consensus state: current_term: 1 leader_uuid: "544cb78f49a847738272a54619aff670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:16.011966 22887 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:16.012071 22887 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "544cb78f49a847738272a54619aff670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:16.012226 22887 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:16.012391 22875 sys_catalog.cc:455] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "544cb78f49a847738272a54619aff670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:16.012493 22875 sys_catalog.cc:458] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:16.018100 22833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "544cb78f49a847738272a54619aff670" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "544cb78f49a847738272a54619aff670" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46673 } } peers { permanent_uuid: "fcd1c0f3673744999377efd91936d7e7" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 46591 } } peers { permanent_uuid: "d0b7a4d95f4c41bcb0effd3f47e9c809" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 40501 } } }
I20260620 00:22:16.018206 22833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670 [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:16.020219 22865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 544cb78f49a847738272a54619aff670: Generated new TSK 0
I20260620 00:22:16.020421 22865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260620 00:22:16.020803 22888 mvcc.cc:205] Tried to move back new op lower bound from 7298723577841172480 to 7298723577652850688. Current Snapshot: MvccSnapshot[applied={T|T < 7298723577662627840}]
W20260620 00:22:16.048343 22678 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.046710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.0.98.62:46673)
0620 00:22:16.046757 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.046760 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.047656 (+ 896us) server_negotiation.cc:407] Connection header received
0620 00:22:16.047757 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.047759 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.047771 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.047801 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.048262 (+ 461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35116: BlockingRecv error: recv got EOF from 127.0.0.1:35116 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 00:22:16.049429 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.047078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38096 (local address 127.0.98.61:46591)
0620 00:22:16.047119 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.047121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.048778 (+ 1657us) server_negotiation.cc:407] Connection header received
0620 00:22:16.048881 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.048883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.048895 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.048924 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.049283 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38096: BlockingRecv error: recv got EOF from 127.0.0.1:38096 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:22:16.050457 22829 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.047382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45358 (local address 127.0.98.60:40501)
0620 00:22:16.047420 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.047422 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.049775 (+ 2353us) server_negotiation.cc:407] Connection header received
0620 00:22:16.049967 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.049969 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.049982 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.050014 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.050394 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45358: BlockingRecv error: recv got EOF from 127.0.0.1:45358 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 00:22:16.053514 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.052552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38104 (local address 127.0.98.61:46591)
0620 00:22:16.052609 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.052611 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.052847 (+ 236us) server_negotiation.cc:407] Connection header received
0620 00:22:16.052974 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.052975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.052995 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.053030 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.053461 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38104: BlockingRecv error: recv got EOF from 127.0.0.1:38104 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260620 00:22:16.057267 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.056267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38108 (local address 127.0.98.61:46591)
0620 00:22:16.056395 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.056397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.056699 (+ 302us) server_negotiation.cc:407] Connection header received
0620 00:22:16.056811 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.056812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.056825 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.056858 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.057215 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38108: BlockingRecv error: recv got EOF from 127.0.0.1:38108 (error 108)
Metrics: {"server-negotiator.queue_time_us":105}
W20260620 00:22:16.061568 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.060685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38122 (local address 127.0.98.61:46591)
0620 00:22:16.060724 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.060726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.060982 (+ 256us) server_negotiation.cc:407] Connection header received
0620 00:22:16.061095 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.061096 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.061108 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.061141 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.061521 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38122: BlockingRecv error: recv got EOF from 127.0.0.1:38122 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 00:22:16.068475 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.067185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38128 (local address 127.0.98.61:46591)
0620 00:22:16.067232 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.067234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.067726 (+ 492us) server_negotiation.cc:407] Connection header received
0620 00:22:16.067873 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.067874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.067889 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.067924 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.068412 (+ 488us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38128: BlockingRecv error: recv got EOF from 127.0.0.1:38128 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 00:22:16.079238 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.078155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38138 (local address 127.0.98.61:46591)
0620 00:22:16.078216 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.078218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.078548 (+ 330us) server_negotiation.cc:407] Connection header received
0620 00:22:16.078674 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.078676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.078689 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.078726 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.079172 (+ 446us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38138: BlockingRecv error: recv got EOF from 127.0.0.1:38138 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
W20260620 00:22:16.098155 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.097060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38146 (local address 127.0.98.61:46591)
0620 00:22:16.097120 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.097122 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.097458 (+ 336us) server_negotiation.cc:407] Connection header received
0620 00:22:16.097586 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.097587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.097604 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.097642 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.098085 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38146: BlockingRecv error: recv got EOF from 127.0.0.1:38146 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260620 00:22:16.132777 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.131831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.0.98.61:46591)
0620 00:22:16.131876 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.131878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.132194 (+ 316us) server_negotiation.cc:407] Connection header received
0620 00:22:16.132298 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.132300 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.132311 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.132339 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.132718 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38154: BlockingRecv error: recv got EOF from 127.0.0.1:38154 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:22:16.199117 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.198332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38168 (local address 127.0.98.61:46591)
0620 00:22:16.198440 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.198443 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.198540 (+ 97us) server_negotiation.cc:407] Connection header received
0620 00:22:16.198570 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.198571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.198582 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.198625 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.199003 (+ 378us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38168: BlockingRecv error: recv got EOF from 127.0.0.1:38168 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260620 00:22:16.329150 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.328340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38178 (local address 127.0.98.61:46591)
0620 00:22:16.328424 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.328426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.328512 (+ 86us) server_negotiation.cc:407] Connection header received
0620 00:22:16.328560 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.328561 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.328571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.328634 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.329065 (+ 431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38178: BlockingRecv error: recv got EOF from 127.0.0.1:38178 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
W20260620 00:22:16.587282 22755 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:16.586491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38182 (local address 127.0.98.61:46591)
0620 00:22:16.586559 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:16.586561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:16.586677 (+ 116us) server_negotiation.cc:407] Connection header received
0620 00:22:16.586758 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:16.586759 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:16.586770 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:16.586843 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:16.587215 (+ 372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38182: BlockingRecv error: recv got EOF from 127.0.0.1:38182 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260620 00:22:16.944725 22857 catalog_manager.cc:1259] Loaded cluster ID: 3458ca1819b447f1b547c374d3e0f69a
I20260620 00:22:16.944806 22857 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809: loading cluster ID for follower catalog manager: success
I20260620 00:22:16.945652 22857 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809: acquiring CA information for follower catalog manager: success
I20260620 00:22:16.945986 22857 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d0b7a4d95f4c41bcb0effd3f47e9c809: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:22:17.006136 22886 catalog_manager.cc:1259] Loaded cluster ID: 3458ca1819b447f1b547c374d3e0f69a
I20260620 00:22:17.006224 22886 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7: loading cluster ID for follower catalog manager: success
I20260620 00:22:17.007120 22886 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7: acquiring CA information for follower catalog manager: success
I20260620 00:22:17.007495 22886 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fcd1c0f3673744999377efd91936d7e7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:22:17.103320 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:17.100570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.0.98.61:46591)
0620 00:22:17.100680 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:17.100682 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:17.100749 (+ 67us) server_negotiation.cc:407] Connection header received
0620 00:22:17.100803 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:17.100804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:17.100859 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:17.100969 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:17.101163 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.101684 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.102499 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.102702 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.102789 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:17.102973 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:17.103084 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:17.103167 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:17.103192 (+ 25us) server_negotiation.cc:299] Negotiation successful
0620 00:22:17.103215 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":49,"threads_started":1}
I20260620 00:22:17.107972 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:17.105656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.0.98.60:40501)
0620 00:22:17.105760 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:17.105762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:17.105770 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:22:17.105805 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:17.105805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:17.105839 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:17.105908 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:17.105992 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.106464 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.107208 (+ 744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.107391 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.107439 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:17.107581 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:17.107685 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:17.107740 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:17.107840 (+ 100us) server_negotiation.cc:299] Negotiation successful
0620 00:22:17.107856 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":52,"threads_started":1}
I20260620 00:22:17.112915 22965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:17.110223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45376 (local address 127.0.98.60:40501)
0620 00:22:17.110262 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:17.110264 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:17.110480 (+ 216us) server_negotiation.cc:407] Connection header received
0620 00:22:17.110580 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:17.110581 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:17.110590 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:17.110652 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:17.110767 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.111249 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.112062 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.112370 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.112433 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:17.112612 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:17.112646 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:17.112703 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:17.112789 (+ 86us) server_negotiation.cc:299] Negotiation successful
0620 00:22:17.112813 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:22:17.112934 22971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:17.109805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35118 (local address 127.0.98.62:46673)
0620 00:22:17.110088 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:17.110091 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:17.110101 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:22:17.110150 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:17.110151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:17.110203 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:17.110296 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:17.110398 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.111179 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.111950 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.112303 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.112379 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:17.112549 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:17.112660 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:17.112751 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:17.112808 (+ 57us) server_negotiation.cc:299] Negotiation successful
0620 00:22:17.112828 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":82,"threads_started":1}
I20260620 00:22:17.113888 22959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:17.110017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38204 (local address 127.0.98.61:46591)
0620 00:22:17.110060 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:17.110062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:17.110140 (+ 78us) server_negotiation.cc:407] Connection header received
0620 00:22:17.110256 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:17.110257 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:17.110271 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:17.110316 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:17.110432 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.111171 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.112907 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:17.113195 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:17.113258 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:17.113610 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:17.113647 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:17.113698 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:17.113783 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 00:22:17.113811 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260620 00:22:17.114836 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 22607
I20260620 00:22:17.121093 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 22682
I20260620 00:22:17.126159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 22757
2026-06-20T00:22:17Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2890 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Jun 20 00:22:17 dist-test-slave-r81b krb5kdc[22976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Jun 20 00:22:17 dist-test-slave-r81b krb5kdc[22976](info): set up 2 sockets
Jun 20 00:22:17 dist-test-slave-r81b krb5kdc[22976](info): commencing operation
krb5kdc: starting...
W20260620 00:22:18.184787 392 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/test-user.keytab.
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-06-20T00:22:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-06-20T00:22:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260620 00:22:18.311012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40433
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40433,127.0.98.61:39851,127.0.98.60:42577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260620 00:22:18.392741 22989 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:22:18.392977 22989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:18.392995 22989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:18.393023 22989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:18.394127 22989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:18.394164 22989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:18.394178 22989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:18.394191 22989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:18.394203 22989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:18.395649 22989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40433,127.0.98.61:39851,127.0.98.60:42577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:18.395929 22989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:18.396198 22989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:18.401371 22994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:18.401400 22995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:18.401396 22997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:18.401469 22989 server_base.cc:1034] running on GCE node
I20260620 00:22:18.401845 22989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:18.402161 22989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:18.403352 22989 hybrid_clock.cc:648] HybridClock initialized: now 1781914938403325 us; error 45 us; skew 500 ppm
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:18.406478 22989 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260620 00:22:18.407058 22989 webserver.cc:460] Webserver started at http://127.0.98.62:37779/ using document root <none> and password file <none>
I20260620 00:22:18.407323 22989 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:18.407408 22989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:18.407567 22989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:18.408725 22989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/data/instance:
uuid: "9995ce8b359e45209d2b9037a65f5d7b"
format_stamp: "Formatted at 2026-06-20 00:22:18 on dist-test-slave-r81b"
server_key: "9a077bf7ac0cc0cb2ce40d321bbb141c"
server_key_iv: "e914e7d959b7878955b6c4e2028c0380"
server_key_version: "encryptionkey@0"
I20260620 00:22:18.409126 22989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/wal/instance:
uuid: "9995ce8b359e45209d2b9037a65f5d7b"
format_stamp: "Formatted at 2026-06-20 00:22:18 on dist-test-slave-r81b"
server_key: "9a077bf7ac0cc0cb2ce40d321bbb141c"
server_key_iv: "e914e7d959b7878955b6c4e2028c0380"
server_key_version: "encryptionkey@0"
I20260620 00:22:18.410538 22989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260620 00:22:18.411401 23004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:18.411675 22989 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:18.411834 22989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/wal
uuid: "9995ce8b359e45209d2b9037a65f5d7b"
format_stamp: "Formatted at 2026-06-20 00:22:18 on dist-test-slave-r81b"
server_key: "9a077bf7ac0cc0cb2ce40d321bbb141c"
server_key_iv: "e914e7d959b7878955b6c4e2028c0380"
server_key_version: "encryptionkey@0"
I20260620 00:22:18.411927 22989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:18.424471 22989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:18.426828 22989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:18.427026 22989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:18.431295 22989 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40433
I20260620 00:22:18.431387 23056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40433 every 8 connection(s)
I20260620 00:22:18.431740 22989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/data/info.pb
I20260620 00:22:18.432828 23057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:18.434422 23057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:22:18.436673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 22989
I20260620 00:22:18.436789 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-0/wal/instance
I20260620 00:22:18.436988 392 external_mini_cluster.cc:1428] Setting key b02d51dd8626eae106ce271831913e36
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
I20260620 00:22:18.446496 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.435219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39074 (local address 127.0.98.62:40433)
0620 00:22:18.435405 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.435415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.435572 (+ 157us) server_negotiation.cc:407] Connection header received
0620 00:22:18.436185 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.436187 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.436204 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.436306 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.436889 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.438267 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.439709 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.440291 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.440422 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.442667 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.442676 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.442677 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.442686 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.444697 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.445099 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.445104 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.445104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.445212 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.445446 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.445448 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.445448 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.445780 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.445860 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.446076 (+ 216us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.446197 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":142,"thread_start_us":87,"threads_started":1}
I20260620 00:22:18.446511 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.435149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40433 (local address 127.0.0.1:39074)
0620 00:22:18.435396 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.435441 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:18.435980 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:18.436310 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:18.436313 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:18.436612 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:18.436767 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.436772 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.438386 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.438387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.439607 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.439611 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.440409 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.440410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.440467 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.441108 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:18.441117 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:18.442571 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:18.444788 (+ 2217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.444794 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.444796 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.445003 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.445292 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.445292 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.445292 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.445379 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.445863 (+ 484us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:18.445869 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:18.445949 (+ 80us) client_negotiation.cc:769] Sending connection context
0620 00:22:18.446041 (+ 92us) client_negotiation.cc:240] Negotiation successful
0620 00:22:18.446265 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":95,"thread_start_us":95,"threads_started":1}
I20260620 00:22:18.447340 23057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:22:18.447783 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.447648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39851 (local address 127.0.0.1:41824)
0620 00:22:18.447714 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.447760 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260620 00:22:18.447938 23005 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
I20260620 00:22:18.448416 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.448317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39851 (local address 127.0.0.1:41834)
0620 00:22:18.448364 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.448392 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:22:18.448522 23057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39851: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260620 00:22:18.463809 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39851
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40433,127.0.98.61:39851,127.0.98.60:42577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:22:18.497783 23057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } attempt: 1
I20260620 00:22:18.498353 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.498197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39851 (local address 127.0.0.1:41844)
0620 00:22:18.498272 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.498318 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260620 00:22:18.498749 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.498650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39851 (local address 127.0.0.1:41856)
0620 00:22:18.498696 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.498723 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260620 00:22:18.498836 23057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39851: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
W20260620 00:22:18.546615 23064 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:22:18.546896 23064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:18.546928 23064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:18.546975 23064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:18.548265 23064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:18.548347 23064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:18.548372 23064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:18.548393 23064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:18.548421 23064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:18.550077 23064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40433,127.0.98.61:39851,127.0.98.60:42577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:39851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:18.550390 23064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:18.550657 23064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:18.555814 23069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:18.555827 23072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:18.556105 23064 server_base.cc:1034] running on GCE node
W20260620 00:22:18.555827 23070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:18.556469 23064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:18.556789 23064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:18.557986 23064 hybrid_clock.cc:648] HybridClock initialized: now 1781914938557981 us; error 38 us; skew 500 ppm
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:18.561143 23064 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260620 00:22:18.561684 23064 webserver.cc:460] Webserver started at http://127.0.98.61:45151/ using document root <none> and password file <none>
I20260620 00:22:18.561952 23064 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:18.562026 23064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:18.562188 23064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:18.563584 23064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/data/instance:
uuid: "584079d2d6584947be5d882f52ac4693"
format_stamp: "Formatted at 2026-06-20 00:22:18 on dist-test-slave-r81b"
server_key: "5a066a24cbd5b569af1285ee788e0e1e"
server_key_iv: "a101317074a60ad4c5016db16943223f"
server_key_version: "encryptionkey@0"
I20260620 00:22:18.564013 23064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/wal/instance:
uuid: "584079d2d6584947be5d882f52ac4693"
format_stamp: "Formatted at 2026-06-20 00:22:18 on dist-test-slave-r81b"
server_key: "5a066a24cbd5b569af1285ee788e0e1e"
server_key_iv: "a101317074a60ad4c5016db16943223f"
server_key_version: "encryptionkey@0"
I20260620 00:22:18.565621 23064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260620 00:22:18.566622 23079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:18.566862 23064 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:18.566985 23064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/wal
uuid: "584079d2d6584947be5d882f52ac4693"
format_stamp: "Formatted at 2026-06-20 00:22:18 on dist-test-slave-r81b"
server_key: "5a066a24cbd5b569af1285ee788e0e1e"
server_key_iv: "a101317074a60ad4c5016db16943223f"
server_key_version: "encryptionkey@0"
I20260620 00:22:18.566998 23057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } attempt: 2
I20260620 00:22:18.567085 23064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:18.567488 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.567294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39851 (local address 127.0.0.1:41868)
0620 00:22:18.567414 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.567453 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260620 00:22:18.567793 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.567698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39851 (local address 127.0.0.1:41884)
0620 00:22:18.567745 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.567769 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:22:18.567885 23057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:39851: Network error: Client connection negotiation failed: client connection to 127.0.98.61:39851: connect: Connection refused (error 111)
I20260620 00:22:18.578181 23064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:18.580595 23064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:18.580771 23064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:18.585284 23064 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:39851
I20260620 00:22:18.585436 23131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:39851 every 8 connection(s)
I20260620 00:22:18.585762 23064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/data/info.pb
I20260620 00:22:18.586872 23132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:18.588598 23132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:22:18.589104 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 23064
I20260620 00:22:18.589224 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-1/wal/instance
I20260620 00:22:18.589444 392 external_mini_cluster.cc:1428] Setting key 702c400ee1ff9f438538afc452a42434
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260620 00:22:18.598829 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.589380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.0.98.62:40433)
0620 00:22:18.589435 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.589438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.589678 (+ 240us) server_negotiation.cc:407] Connection header received
0620 00:22:18.590134 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.590135 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.590151 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.590234 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.590855 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.591350 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.592456 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.592696 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.592773 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.595479 (+ 2706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.595481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.595481 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.595490 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.597304 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.597879 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.597880 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.597881 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.597942 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.598176 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.598177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.598177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.598289 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.598342 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.598684 (+ 342us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.598758 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:22:18.599133 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.589361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40433 (local address 127.0.0.1:39088)
0620 00:22:18.589524 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.589566 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:18.590028 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:18.590288 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:18.590291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:18.590551 (+ 260us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:18.590750 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.590753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.591469 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.591474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.592356 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.592358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.592849 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.592850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.593047 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.593613 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:18.593625 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:18.595366 (+ 1741us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:18.597414 (+ 2048us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.597421 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.597423 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.597800 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.598008 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.598008 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.598009 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.598120 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.598377 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:18.598386 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:18.598552 (+ 166us) client_negotiation.cc:769] Sending connection context
0620 00:22:18.598665 (+ 113us) client_negotiation.cc:240] Negotiation successful
0620 00:22:18.598821 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":91,"threads_started":1}
I20260620 00:22:18.599859 23132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:22:18.608728 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.600308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41898 (local address 127.0.98.61:39851)
0620 00:22:18.600516 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.600521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.600543 (+ 22us) server_negotiation.cc:407] Connection header received
0620 00:22:18.600571 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.600573 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.600583 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.600669 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.600971 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.601744 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.602441 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.602923 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.602990 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.605259 (+ 2269us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.605268 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.605268 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.605275 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.606993 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.607439 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.607444 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.607444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.607508 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.607786 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.607788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.607788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.607977 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.608067 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.608470 (+ 403us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.608625 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":178,"thread_start_us":89,"threads_started":1}
I20260620 00:22:18.608729 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.600263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39851 (local address 127.0.0.1:41898)
0620 00:22:18.600322 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.600334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:18.600394 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:18.600669 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:18.600670 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:18.600817 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:18.600871 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.600873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.601821 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.601822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.602357 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.602360 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.603031 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.603032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.603123 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.603556 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:18.603562 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:18.605161 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:18.607097 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.607098 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.607099 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.607338 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.607591 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.607592 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.607592 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.607661 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.608097 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:18.608098 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:18.608232 (+ 134us) client_negotiation.cc:769] Sending connection context
0620 00:22:18.608416 (+ 184us) client_negotiation.cc:240] Negotiation successful
0620 00:22:18.608567 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260620 00:22:18.609637 23132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:22:18.610069 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.609909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39730)
0620 00:22:18.609968 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.610043 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260620 00:22:18.610344 23080 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
I20260620 00:22:18.610850 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.610762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39736)
0620 00:22:18.610809 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.610833 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260620 00:22:18.610992 23132 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260620 00:22:18.619122 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskbwaPxd/build/release/bin/kudu
/tmp/dist-test-taskbwaPxd/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42577
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:38767
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40433,127.0.98.61:39851,127.0.98.60:42577
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260620 00:22:18.659063 23057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } attempt: 3
I20260620 00:22:18.660233 23132 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } attempt: 1
I20260620 00:22:18.660658 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.660534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39738)
0620 00:22:18.660587 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.660620 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:22:18.661067 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.660977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39752)
0620 00:22:18.661020 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.661044 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260620 00:22:18.661849 23132 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:22:18.666741 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.659518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39851 (local address 127.0.0.1:41900)
0620 00:22:18.659591 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.659604 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:18.659658 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:18.659874 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:18.659875 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:18.660041 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:18.660106 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.660108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.661138 (+ 1030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.661139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.661677 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.661679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.662176 (+ 497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.662177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.662236 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.662680 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:18.662684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:18.663964 (+ 1280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:18.665771 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.665772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.665772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.665948 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.666161 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.666162 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.666162 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.666215 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.666505 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:18.666506 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:18.666546 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 00:22:18.666594 (+ 48us) client_negotiation.cc:240] Negotiation successful
0620 00:22:18.666675 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260620 00:22:18.666805 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.659647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41900 (local address 127.0.98.61:39851)
0620 00:22:18.659719 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.659721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.659731 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:22:18.659761 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.659762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.659775 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.659856 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.660239 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.661033 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.661760 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.662096 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.662169 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.664054 (+ 1885us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.664056 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.664056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.664064 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.665681 (+ 1617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.666028 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.666029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.666030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.666092 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.666276 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.666277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.666277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.666423 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.666503 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.666644 (+ 141us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.666724 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260620 00:22:18.667233 23057 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } has no permanent_uuid. Determining permanent_uuid...
I20260620 00:22:18.667534 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.667446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39762)
0620 00:22:18.667482 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.667512 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260620 00:22:18.667812 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.667761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39776)
0620 00:22:18.667781 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.667800 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260620 00:22:18.667915 23057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
I20260620 00:22:18.699110 23057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } attempt: 1
I20260620 00:22:18.699654 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.699510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39782)
0620 00:22:18.699582 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.699624 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20260620 00:22:18.699978 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.699879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39798)
0620 00:22:18.699927 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.699958 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260620 00:22:18.700099 23057 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
W20260620 00:22:18.705566 23139 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260620 00:22:18.705803 23139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260620 00:22:18.705838 23139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260620 00:22:18.705883 23139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260620 00:22:18.707033 23139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260620 00:22:18.707108 23139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260620 00:22:18.707134 23139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260620 00:22:18.707167 23139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260620 00:22:18.707196 23139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260620 00:22:18.708792 23139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:38767
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40433,127.0.98.61:39851,127.0.98.60:42577
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:42577
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 19 Jun 2026 23:02:49 UTC on ccycloud.quasar-taoxem.root.comops.site
build id 4456768
I20260620 00:22:18.709129 23139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260620 00:22:18.709374 23139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260620 00:22:18.714491 23147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260620 00:22:18.714526 23145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:18.714681 23139 server_base.cc:1034] running on GCE node
W20260620 00:22:18.714491 23144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260620 00:22:18.715081 23139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260620 00:22:18.715431 23139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260620 00:22:18.716629 23139 hybrid_clock.cc:648] HybridClock initialized: now 1781914938716628 us; error 49 us; skew 500 ppm
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260620 00:22:18.719295 23139 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260620 00:22:18.719759 23139 webserver.cc:460] Webserver started at http://127.0.98.60:43711/ using document root <none> and password file <none>
I20260620 00:22:18.720014 23139 fs_manager.cc:359] Metadata directory not provided
I20260620 00:22:18.720062 23139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260620 00:22:18.720233 23139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260620 00:22:18.721340 23139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/data/instance:
uuid: "7437ef5601eb48f1969517ba4ea9e417"
format_stamp: "Formatted at 2026-06-20 00:22:18 on dist-test-slave-r81b"
server_key: "0ae6399af13fd7d0bfa483351567fbb5"
server_key_iv: "4d7683e8bfed89a4f186bda9de2b791c"
server_key_version: "encryptionkey@0"
I20260620 00:22:18.721729 23139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/wal/instance:
uuid: "7437ef5601eb48f1969517ba4ea9e417"
format_stamp: "Formatted at 2026-06-20 00:22:18 on dist-test-slave-r81b"
server_key: "0ae6399af13fd7d0bfa483351567fbb5"
server_key_iv: "4d7683e8bfed89a4f186bda9de2b791c"
server_key_version: "encryptionkey@0"
I20260620 00:22:18.723225 23139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260620 00:22:18.724260 23154 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260620 00:22:18.724553 23139 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260620 00:22:18.724680 23139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/wal
uuid: "7437ef5601eb48f1969517ba4ea9e417"
format_stamp: "Formatted at 2026-06-20 00:22:18 on dist-test-slave-r81b"
server_key: "0ae6399af13fd7d0bfa483351567fbb5"
server_key_iv: "4d7683e8bfed89a4f186bda9de2b791c"
server_key_version: "encryptionkey@0"
I20260620 00:22:18.724767 23139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260620 00:22:18.730051 23132 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } attempt: 2
I20260620 00:22:18.730582 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.730432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39802)
0620 00:22:18.730507 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.730550 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260620 00:22:18.730958 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.730854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39818)
0620 00:22:18.730905 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.730933 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260620 00:22:18.731077 23132 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:42577: Network error: Client connection negotiation failed: client connection to 127.0.98.60:42577: connect: Connection refused (error 111)
I20260620 00:22:18.747296 23139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260620 00:22:18.749594 23139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260620 00:22:18.749752 23139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260620 00:22:18.753928 23139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:42577
I20260620 00:22:18.753978 23206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:42577 every 8 connection(s)
I20260620 00:22:18.754405 23139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/data/info.pb
I20260620 00:22:18.755406 23207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260620 00:22:18.757089 23207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260620 00:22:18.764681 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskbwaPxd/build/release/bin/kudu as pid 23139
I20260620 00:22:18.764825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskbwaPxd/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1781914666918284-392-0/minicluster-data/master-2/wal/instance
I20260620 00:22:18.765029 392 external_mini_cluster.cc:1428] Setting key 20cc13b0db15fdfa958ea91f3f4dd19f
I20260620 00:22:18.767014 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.757838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.0.98.62:40433)
0620 00:22:18.757929 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.757931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.758143 (+ 212us) server_negotiation.cc:407] Connection header received
0620 00:22:18.758671 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.758673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.758685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.758743 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.759381 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.759877 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.760953 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.761166 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.761249 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.763591 (+ 2342us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.763593 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.763593 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.763601 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.765242 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.765875 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.765876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.765877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.765953 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.766267 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.766268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.766268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.766379 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.766436 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.766807 (+ 371us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.766892 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260620 00:22:18.767251 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.757805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40433 (local address 127.0.0.1:39092)
0620 00:22:18.758018 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.758054 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:18.758558 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:18.758821 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:18.758825 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:18.759083 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:18.759284 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.759288 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.759979 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.759980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.760858 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.760860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.761271 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.761272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.761400 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.762010 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:18.762018 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:18.763407 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:18.765385 (+ 1978us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.765393 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.765396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.765794 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.766025 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.766026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.766027 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.766186 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.766453 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:18.766460 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:18.766647 (+ 187us) client_negotiation.cc:769] Sending connection context
0620 00:22:18.766775 (+ 128us) client_negotiation.cc:240] Negotiation successful
0620 00:22:18.766971 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":94,"threads_started":1}
I20260620 00:22:18.768889 23207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260620 00:22:18.775275 23057 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } attempt: 2
I20260620 00:22:18.776363 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.769265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.0.98.61:39851)
0620 00:22:18.769309 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.769311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.769319 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:22:18.769389 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.769390 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.769398 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.769438 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.769783 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.770517 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.771189 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.771471 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.771527 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.773374 (+ 1847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.773376 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.773376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.773385 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.774939 (+ 1554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.775479 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.775480 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.775481 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.775538 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.775757 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.775758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.775758 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.775903 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.775949 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.776177 (+ 228us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.776266 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260620 00:22:18.776392 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.765499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39832 (local address 127.0.98.60:42577)
0620 00:22:18.765660 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.765664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.765718 (+ 54us) server_negotiation.cc:407] Connection header received
0620 00:22:18.765804 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.765806 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.765818 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.765914 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.766785 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.768118 (+ 1333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.769111 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.769538 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.769605 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.772562 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.772572 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.772572 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.772579 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.774567 (+ 1988us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.775159 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.775162 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.775163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.775230 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.775771 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.775771 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.775772 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.775960 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.776119 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.776206 (+ 87us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.776280 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":72,"threads_started":1}
I20260620 00:22:18.776888 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.769218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:39851 (local address 127.0.0.1:41914)
0620 00:22:18.769272 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.769285 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:18.769335 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:18.769450 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:18.769451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:18.769635 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:18.769685 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.769687 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.770603 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.770604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.771101 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.771103 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.771531 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.771532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.771607 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.771993 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:18.771998 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:18.773286 (+ 1288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:18.775006 (+ 1720us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.775007 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.775008 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.775170 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.775617 (+ 447us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.775618 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.775619 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.775680 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.776001 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:18.776002 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:18.776039 (+ 37us) client_negotiation.cc:769] Sending connection context
0620 00:22:18.776691 (+ 652us) client_negotiation.cc:240] Negotiation successful
0620 00:22:18.776788 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260620 00:22:18.777344 23207 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } has no permanent_uuid. Determining permanent_uuid...
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:22:18.783988 23059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.775566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39848)
0620 00:22:18.775625 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.775636 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:18.775673 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:18.775907 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:18.775908 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:18.776084 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:18.776179 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.776182 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.777672 (+ 1490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.777673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.778193 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.778195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.778730 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.778731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.778789 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.779229 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:18.779234 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:18.780696 (+ 1462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:18.782530 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.782532 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.782532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.782760 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.782941 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.782942 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.782942 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.783007 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.783280 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:18.783281 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:18.783327 (+ 46us) client_negotiation.cc:769] Sending connection context
0620 00:22:18.783775 (+ 448us) client_negotiation.cc:240] Negotiation successful
0620 00:22:18.783898 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260620 00:22:18.784008 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.775603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39848 (local address 127.0.98.60:42577)
0620 00:22:18.775783 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.775789 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.775796 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:22:18.775812 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.775812 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.775820 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.775899 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.776263 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.777559 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.778282 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.778648 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.778724 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.780782 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.780783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.780784 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.780790 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.782446 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.782818 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.782819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.782820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.782890 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.783077 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.783078 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.783078 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.783212 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.783277 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.783806 (+ 529us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.783882 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":128,"threads_started":1}
I20260620 00:22:18.785848 23057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b: Bootstrap starting.
I20260620 00:22:18.785948 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.777358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39098 (local address 127.0.98.62:40433)
0620 00:22:18.777526 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.777528 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.777536 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:22:18.777556 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.777557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.777566 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.777618 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.778149 (+ 531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.778798 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.779743 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.779946 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.780021 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.782776 (+ 2755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.782778 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.782778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.782786 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.784496 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.784878 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.784879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.784880 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.784949 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.785183 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.785184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.785185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.785318 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.785484 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.785720 (+ 236us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.785806 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260620 00:22:18.786715 23057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:18.787078 23057 log.cc:824] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:18.787314 23209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.777610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39862)
0620 00:22:18.777647 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.777656 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:18.777687 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:18.778503 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:18.778504 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:18.778676 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:18.778713 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.778715 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.779584 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.779585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.780423 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.780426 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.780939 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.780940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.780990 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.781428 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:18.781433 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:18.783096 (+ 1663us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:18.785770 (+ 2674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.785772 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.785772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.786270 (+ 498us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.786616 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.786617 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.786617 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.786684 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.786992 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:18.786992 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:18.787032 (+ 40us) client_negotiation.cc:769] Sending connection context
0620 00:22:18.787097 (+ 65us) client_negotiation.cc:240] Negotiation successful
0620 00:22:18.787192 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260620 00:22:18.787635 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.778317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.0.98.60:42577)
0620 00:22:18.778387 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.778389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.778397 (+ 8us) server_negotiation.cc:407] Connection header received
0620 00:22:18.778414 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.778415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.778425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.778485 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.778786 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.779497 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.780508 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.780838 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.780938 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.783198 (+ 2260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.783199 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.783199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.783207 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.785684 (+ 2477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.786365 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.786367 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.786367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.786448 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.786748 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.786749 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.786749 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.786904 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.786988 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.787452 (+ 464us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.787548 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260620 00:22:18.787966 23057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b: No bootstrap required, opened a new log
I20260620 00:22:18.789323 23207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417: Bootstrap starting.
I20260620 00:22:18.789999 23207 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:18.790135 23057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } }
I20260620 00:22:18.790253 23057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9995ce8b359e45209d2b9037a65f5d7b, State: Initialized, Role: FOLLOWER
I20260620 00:22:18.790336 23207 log.cc:824] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:18.790338 23057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } }
I20260620 00:22:18.790717 23057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:18.792680 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.792794 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [sys.catalog]: This master's current role is: FOLLOWER
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
W20260620 00:22:18.793082 23226 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:18.793171 23226 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:18.793252 23207 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417: No bootstrap required, opened a new log
I20260620 00:22:18.796118 23207 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } }
I20260620 00:22:18.796327 23207 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7437ef5601eb48f1969517ba4ea9e417, State: Initialized, Role: FOLLOWER
I20260620 00:22:18.796415 23207 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } }
I20260620 00:22:18.793642 22989 master_runner.cc:186] Error getting master registration for 127.0.98.61:39851: OK, instance_id { permanent_uuid: "584079d2d6584947be5d882f52ac4693" instance_seqno: 1781914938583863 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:22:18.796828 22989 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:22:18.796749 23207 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:18.796957 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.797040 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.787027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41922 (local address 127.0.98.61:39851)
0620 00:22:18.787086 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.787088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.787214 (+ 126us) server_negotiation.cc:407] Connection header received
0620 00:22:18.787375 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.787376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.787389 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.787463 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.788115 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.788838 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.789894 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.790239 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.790333 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.793635 (+ 3302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.793637 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.793637 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.793646 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.795418 (+ 1772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.795834 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.795835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.795836 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.795895 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.796094 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.796095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.796095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.796263 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.796438 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.796848 (+ 410us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.796947 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260620 00:22:18.797055 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:18.798502 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260620 00:22:18.799432 23031 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9995ce8b359e45209d2b9037a65f5d7b"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39098
I20260620 00:22:18.799567 23031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260620 00:22:18.799618 23031 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [term 0 FOLLOWER]: Advancing to term 1
W20260620 00:22:18.799647 23239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260620 00:22:18.799711 23239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:18.800611 23031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } }
I20260620 00:22:18.800879 23031 leader_election.cc:288] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [CANDIDATE]: Term 1 election: Requested vote from peers 584079d2d6584947be5d882f52ac4693 (127.0.98.61:39851), 7437ef5601eb48f1969517ba4ea9e417 (127.0.98.60:42577)
I20260620 00:22:18.801224 23181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9995ce8b359e45209d2b9037a65f5d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7437ef5601eb48f1969517ba4ea9e417"
I20260620 00:22:18.801370 23181 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:18.802374 23181 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9995ce8b359e45209d2b9037a65f5d7b in term 1.
I20260620 00:22:18.802804 23005 leader_election.cc:302] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7437ef5601eb48f1969517ba4ea9e417, 9995ce8b359e45209d2b9037a65f5d7b; no voters:
I20260620 00:22:18.803062 23215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [term 1 FOLLOWER]: Leader election won for term 1
I20260620 00:22:18.803252 23215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [term 1 LEADER]: Becoming Leader. State: Replica: 9995ce8b359e45209d2b9037a65f5d7b, State: Running, Role: LEADER
I20260620 00:22:18.803364 23215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } }
I20260620 00:22:18.803998 23139 master_runner.cc:186] Error getting master registration for 127.0.98.61:39851: OK, instance_id { permanent_uuid: "584079d2d6584947be5d882f52ac4693" instance_seqno: 1781914938583863 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260620 00:22:18.806226 23139 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260620 00:22:18.808490 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9995ce8b359e45209d2b9037a65f5d7b. Latest consensus state: current_term: 1 leader_uuid: "9995ce8b359e45209d2b9037a65f5d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.808736 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:18.814821 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.802958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41924 (local address 127.0.98.61:39851)
0620 00:22:18.802999 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.803002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.803011 (+ 9us) server_negotiation.cc:407] Connection header received
0620 00:22:18.803034 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.803034 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.803045 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.803106 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.804422 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.805153 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.807208 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.807538 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.807620 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.810808 (+ 3188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.810810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.810811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.810822 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.812830 (+ 2008us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.813469 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.813470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.813471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.813547 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.813853 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.813855 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.813855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.814032 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.814125 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.814588 (+ 463us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.814694 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260620 00:22:18.817077 23060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.802446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39104 (local address 127.0.98.62:40433)
0620 00:22:18.802495 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.802497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.809122 (+ 6625us) server_negotiation.cc:407] Connection header received
0620 00:22:18.809199 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.809200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.809216 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.809288 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.809997 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.810753 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.811771 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.812078 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.812119 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.813638 (+ 1519us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.813640 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.813640 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.813649 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.815427 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.816223 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.816224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.816224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.816282 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.816472 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.816472 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.816473 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.816596 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.816675 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.816887 (+ 212us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.816972 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260620 00:22:18.819573 23250 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260620 00:22:18.820515 23250 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260620 00:22:18.822224 23132 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } attempt: 3
I20260620 00:22:18.827062 23211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.802765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39878 (local address 127.0.98.60:42577)
0620 00:22:18.802805 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.802807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.814691 (+ 11884us) server_negotiation.cc:407] Connection header received
0620 00:22:18.814808 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.814809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.814822 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.814853 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.815799 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.816557 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.818731 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.819057 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.819100 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.823531 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.823532 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.823533 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.823541 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.825293 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.825930 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.825932 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.825932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.825991 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.826283 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.826284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.826284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.826428 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.826515 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.826862 (+ 347us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.826956 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
Jun 20 00:22:18 dist-test-slave-r81b krb5kdc[22976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1781914938, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260620 00:22:18.830385 23181 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [term 1 FOLLOWER]: Refusing update from remote peer 9995ce8b359e45209d2b9037a65f5d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:22:18.830776 23248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:22:18.832625 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9995ce8b359e45209d2b9037a65f5d7b. Latest consensus state: current_term: 1 leader_uuid: "9995ce8b359e45209d2b9037a65f5d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.832737 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:18.833513 23134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.822498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:42577 (local address 127.0.0.1:39884)
0620 00:22:18.822593 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0620 00:22:18.822604 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0620 00:22:18.822642 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0620 00:22:18.822954 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0620 00:22:18.822955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0620 00:22:18.823129 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0620 00:22:18.823178 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.823180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.824875 (+ 1695us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.824876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.825692 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0620 00:22:18.825695 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.826243 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.826244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0620 00:22:18.826302 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.826719 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0620 00:22:18.826724 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0620 00:22:18.828374 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0620 00:22:18.830763 (+ 2389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.830764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.830764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.830999 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.832804 (+ 1805us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0620 00:22:18.832805 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0620 00:22:18.832806 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0620 00:22:18.832884 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0620 00:22:18.833185 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0620 00:22:18.833186 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0620 00:22:18.833236 (+ 50us) client_negotiation.cc:769] Sending connection context
0620 00:22:18.833302 (+ 66us) client_negotiation.cc:240] Negotiation successful
0620 00:22:18.833409 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260620 00:22:18.833871 23212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:18.822744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39884 (local address 127.0.98.60:42577)
0620 00:22:18.822799 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.822801 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.822808 (+ 7us) server_negotiation.cc:407] Connection header received
0620 00:22:18.822846 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.822846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.822858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.822906 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.823347 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.824756 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.825789 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:18.826125 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:18.826188 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:18.828461 (+ 2273us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0620 00:22:18.828462 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0620 00:22:18.828462 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0620 00:22:18.828470 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0620 00:22:18.830165 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.831086 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.831087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.831088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.831151 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0620 00:22:18.832954 (+ 1803us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0620 00:22:18.832955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0620 00:22:18.832955 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0620 00:22:18.833105 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0620 00:22:18.833176 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:18.833681 (+ 505us) server_negotiation.cc:299] Negotiation successful
0620 00:22:18.833769 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:22:18.835932 23132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693: Bootstrap starting.
I20260620 00:22:18.836791 23132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693: Neither blocks nor log segments found. Creating new log.
I20260620 00:22:18.837111 23132 log.cc:824] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693: Log is configured to *not* fsync() on all Append() calls
I20260620 00:22:18.838019 23248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9995ce8b359e45209d2b9037a65f5d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.838127 23248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:18.838235 23251 mvcc.cc:205] Tried to move back new op lower bound from 7298723589445951488 to 7298723589339340800. Current Snapshot: MvccSnapshot[applied={T|T < 7298723589445951488}]
I20260620 00:22:18.838630 23228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9995ce8b359e45209d2b9037a65f5d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.838734 23228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:18.838935 23252 mvcc.cc:205] Tried to move back new op lower bound from 7298723589445951488 to 7298723589339340800. Current Snapshot: MvccSnapshot[applied={T|T < 7298723589445951488}]
I20260620 00:22:18.839280 23132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693: No bootstrap required, opened a new log
I20260620 00:22:18.841380 23132 raft_consensus.cc:348] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } }
I20260620 00:22:18.841521 23132 raft_consensus.cc:729] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 584079d2d6584947be5d882f52ac4693, State: Initialized, Role: FOLLOWER
I20260620 00:22:18.841600 23132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } }
I20260620 00:22:18.841912 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.841926 23132 sys_catalog.cc:564] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [sys.catalog]: configured and running, proceeding with master startup.
I20260620 00:22:18.842025 23106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9995ce8b359e45209d2b9037a65f5d7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "584079d2d6584947be5d882f52ac4693"
I20260620 00:22:18.842002 23105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [term 0 FOLLOWER]: Advancing to term 1
I20260620 00:22:18.843379 23105 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [term 1 FOLLOWER]: Refusing update from remote peer 9995ce8b359e45209d2b9037a65f5d7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260620 00:22:18.843735 23215 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260620 00:22:18.845256 23250 catalog_manager.cc:1347] Generated new cluster ID: 271fefe4e92040baa1ac23199ec442db
I20260620 00:22:18.845257 23106 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9995ce8b359e45209d2b9037a65f5d7b in term 1.
I20260620 00:22:18.845304 23250 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260620 00:22:18.842307 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:18.846609 23272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9995ce8b359e45209d2b9037a65f5d7b. Latest consensus state: current_term: 1 leader_uuid: "9995ce8b359e45209d2b9037a65f5d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.846680 23257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9995ce8b359e45209d2b9037a65f5d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.846704 23272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:18.846889 23257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693 [sys.catalog]: This master's current role is: FOLLOWER
I20260620 00:22:18.853237 23270 mvcc.cc:205] Tried to move back new op lower bound from 7298723589445951488 to 7298723589339340800. Current Snapshot: MvccSnapshot[applied={T|T < 7298723589445951488}]
I20260620 00:22:18.854372 23268 catalog_manager.cc:1259] Loaded cluster ID: 271fefe4e92040baa1ac23199ec442db
I20260620 00:22:18.854482 23268 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693: loading cluster ID for follower catalog manager: success
W20260620 00:22:18.854611 23268 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260620 00:22:18.854691 23268 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260620 00:22:18.879004 23215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9995ce8b359e45209d2b9037a65f5d7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9995ce8b359e45209d2b9037a65f5d7b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40433 } } peers { permanent_uuid: "584079d2d6584947be5d882f52ac4693" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 39851 } } peers { permanent_uuid: "7437ef5601eb48f1969517ba4ea9e417" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 42577 } } }
I20260620 00:22:18.879168 23215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b [sys.catalog]: This master's current role is: LEADER
I20260620 00:22:18.882941 23250 catalog_manager.cc:1370] Generated new certificate authority record
I20260620 00:22:18.883816 23250 catalog_manager.cc:1504] Loading token signing keys...
I20260620 00:22:18.892208 23250 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9995ce8b359e45209d2b9037a65f5d7b: Generated new TSK 0
I20260620 00:22:18.892426 23250 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260620 00:22:18.941812 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:18.940531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.0.98.61:39851)
0620 00:22:18.940580 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.940582 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.941010 (+ 428us) server_negotiation.cc:407] Connection header received
0620 00:22:18.941127 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.941128 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.941139 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.941170 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.941733 (+ 563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41926: BlockingRecv error: recv got EOF from 127.0.0.1:41926 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 00:22:18.942031 23212 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:18.940805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39886 (local address 127.0.98.60:42577)
0620 00:22:18.940848 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.940850 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.941397 (+ 547us) server_negotiation.cc:407] Connection header received
0620 00:22:18.941526 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.941527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.941538 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.941570 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.941966 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39886: BlockingRecv error: recv got EOF from 127.0.0.1:39886 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260620 00:22:18.942685 23060 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:18.941076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39106 (local address 127.0.98.62:40433)
0620 00:22:18.941121 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.941123 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.942038 (+ 915us) server_negotiation.cc:407] Connection header received
0620 00:22:18.942215 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.942216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.942228 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.942270 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.942624 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39106: BlockingRecv error: recv got EOF from 127.0.0.1:39106 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 00:22:18.950716 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:18.949892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41930 (local address 127.0.98.61:39851)
0620 00:22:18.949933 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.949935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.950137 (+ 202us) server_negotiation.cc:407] Connection header received
0620 00:22:18.950247 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.950248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.950268 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.950299 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.950658 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41930: BlockingRecv error: recv got EOF from 127.0.0.1:41930 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 00:22:18.954020 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:18.953144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41936 (local address 127.0.98.61:39851)
0620 00:22:18.953181 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.953183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.953429 (+ 246us) server_negotiation.cc:407] Connection header received
0620 00:22:18.953535 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.953536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.953547 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.953580 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.953948 (+ 368us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41936: BlockingRecv error: recv got EOF from 127.0.0.1:41936 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 00:22:18.958171 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:18.957269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41946 (local address 127.0.98.61:39851)
0620 00:22:18.957309 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.957310 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.957616 (+ 306us) server_negotiation.cc:407] Connection header received
0620 00:22:18.957715 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.957716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.957728 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.957758 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.958114 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41946: BlockingRecv error: recv got EOF from 127.0.0.1:41946 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260620 00:22:18.964576 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:18.963486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41952 (local address 127.0.98.61:39851)
0620 00:22:18.963527 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.963534 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.963926 (+ 392us) server_negotiation.cc:407] Connection header received
0620 00:22:18.964050 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.964051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.964063 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.964093 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.964516 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41952: BlockingRecv error: recv got EOF from 127.0.0.1:41952 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260620 00:22:18.974855 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:18.973927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41966 (local address 127.0.98.61:39851)
0620 00:22:18.973973 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.973975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.974241 (+ 266us) server_negotiation.cc:407] Connection header received
0620 00:22:18.974350 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.974351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.974364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.974396 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.974794 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41966: BlockingRecv error: recv got EOF from 127.0.0.1:41966 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260620 00:22:18.993854 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:18.992843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41980 (local address 127.0.98.61:39851)
0620 00:22:18.992892 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:18.992895 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:18.993239 (+ 344us) server_negotiation.cc:407] Connection header received
0620 00:22:18.993350 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:18.993351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:18.993364 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:18.993395 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:18.993786 (+ 391us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41980: BlockingRecv error: recv got EOF from 127.0.0.1:41980 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260620 00:22:19.028303 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:19.027454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41988 (local address 127.0.98.61:39851)
0620 00:22:19.027526 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:19.027529 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:19.027691 (+ 162us) server_negotiation.cc:407] Connection header received
0620 00:22:19.027727 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:19.027728 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:19.027742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:19.027822 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:19.028239 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41988: BlockingRecv error: recv got EOF from 127.0.0.1:41988 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
W20260620 00:22:19.094491 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:19.093653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41990 (local address 127.0.98.61:39851)
0620 00:22:19.093724 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:19.093727 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:19.093890 (+ 163us) server_negotiation.cc:407] Connection header received
0620 00:22:19.093925 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:19.093926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:19.093938 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:19.094007 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:19.094413 (+ 406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41990: BlockingRecv error: recv got EOF from 127.0.0.1:41990 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260620 00:22:19.225072 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:19.224011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41998 (local address 127.0.98.61:39851)
0620 00:22:19.224082 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:19.224084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:19.224214 (+ 130us) server_negotiation.cc:407] Connection header received
0620 00:22:19.224350 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:19.224351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:19.224366 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:19.224410 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:19.225001 (+ 591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41998: BlockingRecv error: recv got EOF from 127.0.0.1:41998 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
W20260620 00:22:19.483294 23136 negotiation.cc:337] Failed RPC negotiation. Trace:
0620 00:22:19.482477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42010 (local address 127.0.98.61:39851)
0620 00:22:19.482569 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:19.482571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:19.482584 (+ 13us) server_negotiation.cc:407] Connection header received
0620 00:22:19.482661 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:19.482662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:19.482672 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:19.482743 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0620 00:22:19.483236 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42010: BlockingRecv error: recv got EOF from 127.0.0.1:42010 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20260620 00:22:19.800294 23239 catalog_manager.cc:1259] Loaded cluster ID: 271fefe4e92040baa1ac23199ec442db
I20260620 00:22:19.800392 23239 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417: loading cluster ID for follower catalog manager: success
I20260620 00:22:19.801234 23239 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417: acquiring CA information for follower catalog manager: success
I20260620 00:22:19.801549 23239 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7437ef5601eb48f1969517ba4ea9e417: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:22:19.855851 23268 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693: acquiring CA information for follower catalog manager: success
I20260620 00:22:19.856287 23268 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 584079d2d6584947be5d882f52ac4693: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260620 00:22:19.999897 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:19.996680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.0.98.61:39851)
0620 00:22:19.996903 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:19.996905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:19.996916 (+ 11us) server_negotiation.cc:407] Connection header received
0620 00:22:19.996943 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:19.996943 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:19.996997 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:19.997082 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:19.997309 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:19.997992 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:19.998870 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:19.999080 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:19.999139 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:19.999491 (+ 352us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:19.999634 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:19.999725 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:19.999794 (+ 69us) server_negotiation.cc:299] Negotiation successful
0620 00:22:19.999820 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191,"thread_start_us":69,"threads_started":1}
I20260620 00:22:20.004909 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:20.002212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39898 (local address 127.0.98.60:42577)
0620 00:22:20.002308 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:20.002310 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:20.002353 (+ 43us) server_negotiation.cc:407] Connection header received
0620 00:22:20.002429 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:20.002430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:20.002464 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:20.002535 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:20.002627 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:20.003137 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:20.003915 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:20.004180 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:20.004287 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:20.004486 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:20.004623 (+ 137us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:20.004688 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:20.004770 (+ 82us) server_negotiation.cc:299] Negotiation successful
0620 00:22:20.004789 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"thread_start_us":55,"threads_started":1}
I20260620 00:22:20.010178 23345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:20.007302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39912 (local address 127.0.98.60:42577)
0620 00:22:20.007343 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:20.007345 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:20.007407 (+ 62us) server_negotiation.cc:407] Connection header received
0620 00:22:20.007530 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:20.007531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:20.007546 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:20.007620 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:20.007752 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:20.008539 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:20.009407 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:20.009643 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:20.009709 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:20.009869 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:20.009905 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:20.009962 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:20.010047 (+ 85us) server_negotiation.cc:299] Negotiation successful
0620 00:22:20.010071 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260620 00:22:20.010499 23339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:20.007051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42020 (local address 127.0.98.61:39851)
0620 00:22:20.007107 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:20.007110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:20.007382 (+ 272us) server_negotiation.cc:407] Connection header received
0620 00:22:20.007450 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:20.007451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:20.007465 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:20.007569 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:20.007792 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:20.008582 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:20.009639 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:20.009917 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:20.009978 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:20.010197 (+ 219us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:20.010233 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:20.010271 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:20.010390 (+ 119us) server_negotiation.cc:299] Negotiation successful
0620 00:22:20.010421 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260620 00:22:20.010684 23352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0620 00:22:20.007011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39122 (local address 127.0.98.62:40433)
0620 00:22:20.007179 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0620 00:22:20.007181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0620 00:22:20.007191 (+ 10us) server_negotiation.cc:407] Connection header received
0620 00:22:20.007305 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0620 00:22:20.007306 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0620 00:22:20.007369 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0620 00:22:20.007459 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0620 00:22:20.007651 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:20.008327 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:20.009482 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0620 00:22:20.009827 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0620 00:22:20.009893 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0620 00:22:20.010170 (+ 277us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0620 00:22:20.010310 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0620 00:22:20.010382 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0620 00:22:20.010557 (+ 175us) server_negotiation.cc:299] Negotiation successful
0620 00:22:20.010585 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":61,"threads_started":1}
I20260620 00:22:20.012770 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 22989
I20260620 00:22:20.018718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 23064
I20260620 00:22:20.023749 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskbwaPxd/build/release/bin/kudu with pid 23139
2026-06-20T00:22:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2898 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5788 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (273111 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST